Image signal processor front-end image data processing system and method

ABSTRACT

Various techniques are provided herein for processing raw image data in front-end processing logic of an image signal processing system. In one embodiment, the front-end processing logic includes a statistics processing unit configured to process raw image data acquired by an image sensor to obtain one or more sets of statistics. The statistics processing unit may first correct defective pixels in the raw image data and then correct lens shading errors in the raw image data prior to extracting the statistics information. In certain embodiments, black level compensation may be applied between the defective pixel correction and lens shading correction steps, and inverse black level compensation may be applied between the lens shading correction step and the extraction of the statistics information. The acquired statistics information may be utilized by an image signal processing pipeline for converting the raw image data into a color (e.g., RGB) and/or luma (e.g., YCbCr) image.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 12/582,414 filed on Oct. 20, 2009.

BACKGROUND

The present disclosure relates generally to digital imaging devices and,more particularly, to systems and method for processing image dataobtained using an image sensor of a digital imaging device.

This section is intended to introduce the reader to various aspects ofart that may be related to various aspects of the present techniques,which are described and/or claimed below. This discussion is believed tobe helpful in providing the reader with background information tofacilitate a better understanding of the various aspects of the presentdisclosure. Accordingly, it should be understood that these statementsare to be read in this light, and not as admissions of prior art.

In recent years, digital imaging devices have become increasing populardue, at least in part, to such devices becoming more and more affordablefor the average consumer. Further, in addition to a number ofstand-alone digital cameras currently available on the market, it is notuncommon for digital imaging devices to be integrated as part of anotherelectronic device, such as a desktop or notebook computer, a cellularphone, or a portable media player.

To acquire image data, most digital imaging devices include an imagesensor that provides a number of light-detecting elements (e.g.,photodetectors) configured to convert light detected by the image sensorinto an electrical signal. An image sensor may also include a colorfilter array that filters light captured by the image sensor to capturecolor information. The image data captured by the image sensor may thenbe processed by an image processing pipeline, which may apply a numberof various image processing operations to the image data to generate afull color image that may be displayed for viewing on a display device,such as a monitor.

While conventional image processing techniques generally aim to producea viewable image that is both objectively and subjectively pleasing to aviewer, such conventional techniques may not adequately address errorsand/or distortions in the image data introduced by the imaging deviceand/or the image sensor. For instance, defective pixels on the imagesensor, which may be due to manufacturing defects or operationalfailure, may fail to sense light levels accurately and, if notcorrected, may manifest as artifacts appearing in the resultingprocessed image. Additionally, light intensity fall-off at the edges ofthe image sensor, which may be due to imperfections in the manufactureof the lens, may adversely affect characterization measurements and mayresult in an image in which the overall light intensity is non-uniform.The image processing pipeline may also perform one or more processes tosharpen the image. Conventional sharpening techniques, however, may notadequately account for existing noise in the image signal, or may beunable to distinguish the noise from edges and textured areas in theimage. In such instances, conventional sharpening techniques mayactually increase the appearance of noise in the image, which isgenerally undesirable.

Another image processing operation that may be applied to the image datacaptured by the image sensor is a demosaicing operation. Because thecolor filter array generally provides color data at one wavelength persensor pixel, a full set of color data is generally interpolated foreach color channel in order to reproduce a full color image (e.g., RGBimage). Conventional demosaicing techniques generally interpolate valuesfor the missing color data in a horizontal or a vertical direction,generally depending on some type of fixed threshold. However, suchconventional demosaicing techniques may not adequately account for thelocations and direction of edges within the image, which may result inedge artifacts, such as aliasing, checkerboard artifacts, or rainbowartifacts, being introduced into the full color image, particularlyalong diagonal edges within the image.

Accordingly, various considerations should be addressed when processinga digital image obtained with a digital camera or other imaging devicein order to improve the appearance of the resulting image. Inparticular, certain aspects of the disclosure below may address one ormore of the drawbacks briefly mentioned above.

SUMMARY

A summary of certain embodiments disclosed herein is set forth below. Itshould be understood that these aspects are presented merely to providethe reader with a brief summary of these certain embodiments and thatthese aspects are not intended to limit the scope of this disclosure.Indeed, this disclosure may encompass a variety of aspects that may notbe set forth below.

The present disclosure provides various techniques for processing imagedata acquired using a digital image sensor. In accordance with aspectsof the present disclosure, one such technique may relate to theprocessing of raw image data in front-end processing logic of an imagesignal processor system. In accordance with one embodiment, thefront-end processing logic includes at least one statistics processingunits and at least one pixel processing unit. The pixel processing unitmay receive raw image data acquired by a digital image sensor and mayprocess the raw image data to reduce noise and to scale the image data.The raw image data processed by the pixel processing unit may then beprovided either directly (e.g., using one or more data queues, such asFIFO queues) or through system memory to an image signal processingpipeline (also part of the image signal processor system) locateddownstream from the front-end processing logic.

The statistics processing unit may be configured to receive raw imagedata acquired using the digital image sensor and to process the rawimage data to acquire one or more sets of statistics data pertaining toproperties of the image data and/or the image sensor. In one embodiment,the statistics processing unit may be configured to first process theraw image data to detect and correct defective pixels. The statisticsprocessing unit may then process the raw image data to correct for lensshading distortions in the raw image data. Thereafter, the statisticprocessing unit may process the raw image data to acquire one or moresets of statistics data, which may include auto-exposure statistics,auto-focus statistics, and/or auto-white balance statistics information.In some embodiments, the statistics processing unit may be configured toapply a black level compensation operation to the raw image datasubsequent to the correcting of defective pixels, but prior to thecorrecting of lens shading distortion, and to apply an inverse blacklevel compensation operation to the raw image data subsequent to thecorrecting of lens shading distortions but prior to the acquisition ofthe statistics data. Using this technique, the statistics informationmay be utilized by the above-mentioned image processing pipeline duringthe conversion of the raw image data into an RGB image and/or into aluma image.

Various refinements of the features noted above may exist in relation tovarious aspects of the present disclosure. Further features may also beincorporated in these various aspects as well. These refinements andadditional features may exist individually or in any combination. Forinstance, various features discussed below in relation to one or more ofthe illustrated embodiments may be incorporated into any of theabove-described aspects of the present disclosure alone or in anycombination. Again, the brief summary presented above is intended onlyto familiarize the reader with certain aspects and contexts ofembodiments of the present disclosure without limitation to the claimedsubject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent of application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings will be provided by the Office upon request and paymentof the necessary fee.

Various aspects of this disclosure may be better understood upon readingthe following detailed description and upon reference to the drawings inwhich:

FIG. 1 is a simplified block diagram depicting components of an exampleof an electronic device that includes an imaging device and imageprocessing circuitry configured to implement one or more of the imageprocessing technique set forth in the present disclosure;

FIG. 2 shows a graphical representation of a 2×2 pixel block of a Bayercolor filter array that may be implemented in the imaging device of FIG.1;

FIG. 3 is a perspective view of the electronic device of FIG. 1 in theform of a laptop computing device, in accordance with aspects of thepresent disclosure;

FIG. 4 is a front view of the electronic device of FIG. 1 in the form ofa desktop computing device, in accordance with aspects of the presentdisclosure;

FIG. 5 is a front view of the electronic device of FIG. 1 in the form ofa handheld portable electronic device, in accordance with aspects of thepresent disclosure;

FIG. 6 is a rear view of the electronic device shown in FIG. 5;

FIG. 7 is a block diagram illustrating front-end image signal processing(ISP) logic and ISP pipe processing logic that may be implemented in theimage processing circuitry of FIG. 1, in accordance with aspects of thepresent disclosure;

FIG. 8 is a more detailed block diagram showing an embodiment of the ISPfront-end logic of FIG. 7, in accordance with aspects of the presentdisclosure;

FIG. 9 is graphical depiction of various imaging regions that may bedefined within a source image frame captured by an image sensor, inaccordance with aspects of the present disclosure;

FIG. 10 is a block diagram that provides a more detailed view of oneembodiment of the ISP front-end pixel processing unit, as shown in theISP front-end logic of FIG. 8, in accordance with aspects of the presentdisclosure;

FIG. 11 is a process diagram illustrating how temporal filtering may beapplied to image pixel data received by the ISP front-end pixelprocessing unit shown in FIG. 10, in accordance with one embodiment;

FIG. 12 illustrates a set of reference image pixels and a set ofcorresponding current image pixels that may be used to determine one ormore parameters for the temporal filtering process shown in FIG. 11;

FIG. 13 is a more detailed view showing one embodiment of a binningcompensation filter that may be implemented in the ISP front-end pixelprocessing unit of FIG. 10, in accordance with aspects of the presentdisclosure;

FIG. 14 illustrates a process for scaling image data using the binningcompensation filter of FIG. 13, in accordance with one embodiment;

FIG. 15 is more a more detailed block diagram showing an embodiment of astatistics processing unit which may be implemented in the ISP front-endprocessing logic, as shown in FIG. 8, in accordance with aspects of thepresent disclosure;

FIG. 16 shows various image frame boundary cases that may be consideredwhen applying techniques for detecting and correcting defective pixelsduring statistics processing by the statistics processing unit of FIG.15, in accordance with aspects of the present disclosure;

FIG. 17 is a flow chart illustrating a process for performing defectivepixel detection and correction during statistics processing, inaccordance with one embodiment;

FIG. 18 shows a three-dimensional profile depicting light intensityversus pixel position for a conventional lens of an imaging device;

FIG. 19 is a colored drawing that exhibits non-uniform light intensityacross the image, which may be the result of lens shadingirregularities;

FIG. 20 is a graphical illustration of a raw imaging frame that includesa lens shading correction region and a gain grid, in accordance withaspects of the present disclosure;

FIG. 21 illustrates the interpolation of a gain value for an image pixelenclosed by four bordering grid gain points, in accordance with aspectsof the present disclosure;

FIG. 22 is a flow chart illustrating a process for determininginterpolated gain values that may be applied to imaging pixels during alens shading correction operation, in accordance with an embodiment ofthe present technique;

FIG. 23 is a three-dimensional profile depicting interpolated gainvalues that may be applied to an image that exhibits the light intensitycharacteristics shown in FIG. 18 when performing lens shadingcorrection, in accordance with aspects of the present disclosure;

FIG. 24 shows the colored drawing from FIG. 19 that exhibits improveduniformity in light intensity after a lens shading correction operationis applied, in accordance with accordance aspects of the presentdisclosure;

FIG. 25 graphically illustrates how a radial distance between a currentpixel and the center of an image may be calculated and used to determinea radial gain component for lens shading correction, in accordance withone embodiment;

FIG. 26 is a flow chart illustrating a process by which radial gains andinterpolated gains from a gain grid are used to determine a total gainthat may be applied to imaging pixels during a lens shading correctionoperation, in accordance with an embodiment of the present technique;

FIG. 27 is a block diagram showing an embodiment of the ISP pipeprocessing logic of FIG. 7, in accordance with aspects of the presentdisclosure;

FIG. 28 is a more detailed view showing an embodiment of a raw pixelprocessing block that may be implemented in the ISP pipe processinglogic of FIG. 27, in accordance with aspects of the present disclosure;

FIG. 29 shows various image frame boundary cases that may be consideredwhen applying techniques for detecting and correcting defective pixelsduring processing by the raw pixel processing block shown in FIG. 28, inaccordance with aspects of the present disclosure;

FIGS. 30-32 are flowcharts that depict various processes for detectingand correcting defective pixels that may be performed in the raw pixelprocessing block of FIG. 28, in accordance with one embodiment;

FIG. 33 shows the location of two green pixels in a 2×2 pixel block of aBayer image sensor that may be interpolated when applying greennon-uniformity correction techniques during processing by the raw pixelprocessing logic of FIG. 28, in accordance with aspects of the presentdisclosure;

FIG. 34 illustrates a set of pixels that includes a center pixel andassociated horizontal neighboring pixels that may be used as part of ahorizontal filtering process for noise reduction, in accordance withaspects of the present disclosure;

FIG. 35 illustrates a set of pixels that includes a center pixel andassociated vertical neighboring pixels that may be used as part of avertical filtering process for noise reduction, in accordance withaspects of the present disclosure;

FIG. 36 is a simplified flow diagram that depicts how demosaicing may beapplied to a raw Bayer image pattern to produce a full color RGB image;

FIG. 37 depicts a set of pixels of a Bayer image pattern from whichhorizontal and vertical energy components may be derived forinterpolating green color values during demosaicing of the Bayer imagepattern, in accordance with one embodiment;

FIG. 38 shows a set of horizontal pixels to which filtering may beapplied to determine a horizontal component of an interpolated greencolor value during demosaicing of a Bayer image pattern, in accordancewith aspects of the present technique;

FIG. 39 shows a set of vertical pixels to which filtering may be appliedto determine a vertical component of an interpolated green color valueduring demosaicing of a Bayer image pattern, in accordance with aspectsof the present technique;

FIG. 40 shows various 3×3 pixel blocks to which filtering may be appliedto determine interpolated red and blue values during demosaicing of aBayer image pattern, in accordance with aspects of the presenttechnique;

FIGS. 41-44 provide flowcharts that depict various processes forinterpolating green, red, and blue color values during demosaicing of aBayer image pattern, in accordance with one embodiment;

FIG. 45 shows a colored drawing of an original image scene that may becaptured by an image sensor and processed in accordance with aspects ofthe demosaicing techniques disclosed herein;

FIG. 46 shows a colored drawing of Bayer image pattern of the imagescene shown in FIG. 45;

FIG. 47 shows a colored drawing of an RGB image reconstructed using aconventional demosaicing technique based upon the Bayer image pattern ofFIG. 46;

FIG. 48 shows a colored drawing of an RGB image reconstructed from theBayer image pattern of FIG. 46 in accordance with aspects of thedemosaicing techniques disclosed herein;

FIG. 49 is a more detailed view showing one embodiment of an RGBprocessing block that may be implemented in the ISP pipe processinglogic of FIG. 27, in accordance with aspects of the present disclosure;

FIG. 50 is a more detailed view showing one embodiment of a YCbCrprocessing block that may be implemented in the ISP pipe processinglogic of FIG. 27, in accordance with aspects of the present disclosure;

FIG. 51 is a graphical depiction of active source regions for luma andchroma, as defined within a source buffer using a 1-plane format, inaccordance with aspects of the present disclosure;

FIG. 52 is a graphical depiction of active source regions for luma andchroma, as defined within a source buffer using a 2-plane format, inaccordance with aspects of the present disclosure;

FIG. 53 is a block diagram illustrating image sharpening logic that maybe implemented in the YCbCr processing block, as shown in FIG. 50, inaccordance with one embodiment;

FIG. 54 is a block diagram illustrating edge enhancement logic that maybe implemented in the YCbCr processing block, as shown in FIG. 50, inaccordance with one embodiment;

FIG. 55 is a graph showing the relationship of chroma attenuationfactors to sharpened luma values, in accordance with aspects of thepresent disclosure;

FIG. 56 is a block diagram illustrating image brightness, contrast, andcolor (BCC) adjustment logic that may be implemented in the YCbCrprocessing block, as shown in FIG. 50, in accordance with oneembodiment; and

FIG. 57 shows a hue and saturation color wheel in the YCbCr color spacedefining various hue angles and saturation values that may be appliedduring color adjustment in the BCC adjustment logic shown in FIG. 56.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

One or more specific embodiments of the present disclosure will bedescribed below. These described embodiments are only examples of thepresently disclosed techniques. Additionally, in an effort to provide aconcise description of these embodiments, all features of an actualimplementation may not be described in the specification. It should beappreciated that in the development of any such actual implementation,as in any engineering or design project, numerousimplementation-specific decisions must be made to achieve thedevelopers' specific goals, such as compliance with system-related andbusiness-related constraints, which may vary from one implementation toanother. Moreover, it should be appreciated that such a developmenteffort might be complex and time consuming, but would nevertheless be aroutine undertaking of design, fabrication, and manufacture for those ofordinary skill having the benefit of this disclosure.

When introducing elements of various embodiments of the presentdisclosure, the articles “a,” “an,” and “the” are intended to mean thatthere are one or more of the elements. The terms “comprising,”“including,” and “having” are intended to be inclusive and mean thatthere may be additional elements other than the listed elements.Additionally, it should be understood that references to “oneembodiment” or “an embodiment” of the present disclosure are notintended to be interpreted as excluding the existence of additionalembodiments that also incorporate the recited features.

As will be discussed below, the present disclosure relates generally totechniques for processing image data acquired via one or more imagesensing devices. In particular, certain aspects of the presentdisclosure may relate to techniques for detecting and correctingdefective pixels, techniques for demosaicing a raw image pattern,techniques for sharpening a luminance image using a multi-scale unsharpmask, and techniques for applying lens shading gains to correct for lensshading irregularities. Further, it should be understood that thepresently disclosed techniques may be applied to both still images andmoving images (e.g., video), and may be utilized in any suitable type ofimaging application, such as a digital camera, an electronic devicehaving an integrated digital camera, a security or video surveillancesystem, a medical imaging system, and so forth.

Keeping the above points in mind, FIG. 1 is a block diagram illustratingan example of an electronic device 10 that may provide for theprocessing of image data using one or more of the image processingtechniques briefly mentioned above. The electronic device 10 may be anytype of electronic device, such as a laptop or desktop computer, amobile phone, a digital media player, or the like, that is configured toreceive and process image data, such as data acquired using one or moreimage sensing components. By way of example only, the electronic device10 may be a portable electronic device, such as a model of an iPod® oriPhone®, available from Apple Inc. of Cupertino, Calif. Additionally,the electronic device 10 may be a desktop or laptop computer, such as amodel of a MacBook®, MacBook® Pro, MacBook Air®, iMac®, Mac® Mini, orMac Pro®, available from Apple Inc. In other embodiments, electronicdevice 10 may also be a model of an electronic device from anothermanufacturer that is capable of acquiring and processing image data.

Regardless of its form (e.g., portable or non-portable), it should beunderstood that the electronic device 10 may provide for the processingof image data using one or more of the image processing techniquesbriefly discussed above, which may include defective pixel correctionand/or detection techniques, lens shading correction techniques,demosaicing techniques, or image sharpening techniques, among others. Insome embodiments, the electronic device 10 may apply such imageprocessing techniques to image data stored in a memory of the electronicdevice 10. In further embodiments, the electronic device 10 may includeone or more imaging devices, such as an integrated or external digitalcamera, configured to acquire image data, which may then be processed bythe electronic device 10 using one or more of the above-mentioned imageprocessing techniques. Embodiments showing both portable andnon-portable embodiments of electronic device 10 will be furtherdiscussed below in FIGS. 3-6.

As shown in FIG. 1, the electronic device 10 may include variousinternal and/or external components which contribute to the function ofthe device 10. Those of ordinary skill in the art will appreciate thatthe various functional blocks shown in FIG. 1 may comprise hardwareelements (including circuitry), software elements (including computercode stored on a computer-readable medium) or a combination of bothhardware and software elements. For example, in the presentlyillustrated embodiment, the electronic device 10 may includeinput/output (I/O) ports 12, input structures 14, one or more processors16, memory device 18, non-volatile storage 20, expansion card(s) 22,networking device 24, power source 26, and display 28. Additionally, theelectronic device 10 may include one or more imaging devices 30, such asa digital camera, and image processing circuitry 32. As will bediscussed further below, the image processing circuitry 32 may beconfigured implement one or more of the above-discussed image processingtechniques when processing image data. As can be appreciated, image dataprocessed by image processing circuitry 32 may be retrieved from thememory 18 and/or the non-volatile storage device(s) 20, or may beacquired using the imaging device 30.

Before continuing, it should be understood that the system block diagramof the device 10 shown in FIG. 1 is intended to be a high-level controldiagram depicting various components that may be included in such adevice 10. That is, the connection lines between each individualcomponent shown in FIG. 1 may not necessarily represent paths ordirections through which data flows or is transmitted between variouscomponents of the device 10. Indeed, as discussed below, the depictedprocessor(s) 16 may, in some embodiments, include multiple processors,such as a main processor (e.g., CPU), and dedicated image and/or videoprocessors. In such embodiments, the processing of image data may beprimarily handled by these dedicated processors, thus effectivelyoffloading such tasks from a main processor (CPU).

With regard to each of the illustrated components in FIG. 1, the I/Oports 12 may include ports configured to connect to a variety ofexternal devices, such as a power source, an audio output device (e.g.,headset or headphones), or other electronic devices (such as handhelddevices and/or computers, printers, projectors, external displays,modems, docking stations, and so forth). In one embodiment, the I/Oports 12 may be configured to connect to an external imaging device,such as a digital camera, for the acquisition of image data that may beprocessed using the image processing circuitry 32. The I/O ports 12 maysupport any suitable interface type, such as a universal serial bus(USB) port, a serial connection port, an IEEE-1394 (FireWire) port, anEthernet or modem port, and/or an AC/DC power connection port.

In some embodiments, certain I/O ports 12 may be configured to providefor more than one function. For instance, in one embodiment, the I/Oports 12 may include a proprietary port from Apple Inc. that mayfunction not only to facilitate the transfer of data between theelectronic device 10 and an external source, but also to couple thedevice 10 to a power charging interface such as an power adapterdesigned to provide power from a electrical wall outlet, or an interfacecable configured to draw power from another electrical device, such as adesktop or laptop computer, for charging the power source 26 (which mayinclude one or more rechargeable batteries). Thus, the I/O port 12 maybe configured to function dually as both a data transfer port and anAC/DC power connection port depending, for example, on the externalcomponent being coupled to the device 10 via the I/O port 12.

The input structures 14 may provide user input or feedback to theprocessor(s) 16. For instance, input structures 14 may be configured tocontrol one or more functions of electronic device 10, such asapplications running on electronic device 10. By way of example only,input structures 14 may include buttons, sliders, switches, controlpads, keys, knobs, scroll wheels, keyboards, mice, touchpads, and soforth, or some combination thereof. In one embodiment, input structures14 may allow a user to navigate a graphical user interface (GUI)displayed on device 10. Additionally, input structures 14 may include atouch sensitive mechanism provided in conjunction with display 28. Insuch embodiments, a user may select or interact with displayed interfaceelements via the touch sensitive mechanism.

The input structures 14 may include the various devices, circuitry, andpathways by which user input or feedback is provided to one or moreprocessors 16. Such input structures 14 may be configured to control afunction of the device 10, applications running on the device 10, and/orany interfaces or devices connected to or used by the electronic device10. For example, the input structures 14 may allow a user to navigate adisplayed user interface or application interface. Examples of the inputstructures 14 may include buttons, sliders, switches, control pads,keys, knobs, scroll wheels, keyboards, mice, touchpads, and so forth.

In certain embodiments, an input structure 14 and the display device 28may be provided together, such as in the case of a “touchscreen,”whereby a touch-sensitive mechanism is provided in conjunction with thedisplay 28. In such embodiments, the user may select or interact withdisplayed interface elements via the touch-sensitive mechanism. In thisway, the displayed interface may provide interactive functionality,allowing a user to navigate the displayed interface by touching thedisplay 28. For example, user interaction with the input structures 14,such as to interact with a user or application interface displayed onthe display 26, may generate electrical signals indicative of the userinput. These input signals may be routed via suitable pathways, such asan input hub or data bus, to the one or more processors 16 for furtherprocessing.

In addition to processing various input signals received via the inputstructure(s) 14, the processor(s) 16 may control the general operationof the device 10. For instance, the processor(s) 16 may provide theprocessing capability to execute an operating system, programs, user andapplication interfaces, and any other functions of the electronic device10. The processor(s) 16 may include one or more microprocessors, such asone or more “general-purpose” microprocessors, one or morespecial-purpose microprocessors and/or application-specificmicroprocessors (ASICs), or a combination of such processing components.For example, the processor(s) 16 may include one or more instruction set(e.g., RISC) processors, as well as graphics processors (GPU), videoprocessors, audio processors and/or related chip sets. As will beappreciated, the processor(s) 16 may be coupled to one or more databuses for transferring data and instructions between various componentsof the device 10. In certain embodiments, the processor(s) 16 mayprovide the processing capability to execute an imaging applications onthe electronic device 10, such as Photo Booth®, Aperture®, iPhoto®, orPreview®, available from Apple Inc., or the “Camera” and/or “Photo”applications provided by Apple Inc. and available on models of theiPhone®.

The instructions or data to be processed by the processor(s) 16 may bestored in a computer-readable medium, such as a memory device 18. Thememory device 18 may be provided as a volatile memory, such as randomaccess memory (RAM) or as a non-volatile memory, such as read-onlymemory (ROM), or as a combination of one or more RAM and ROM devices.The memory 18 may store a variety of information and may be used forvarious purposes. For example, the memory 18 may store firmware for theelectronic device 10, such as a basic input/output system (BIOS), anoperating system, various programs, applications, or any other routinesthat may be executed on the electronic device 10, including userinterface functions, processor functions, and so forth. In addition, thememory 18 may be used for buffering or caching during operation of theelectronic device 10. For instance, in one embodiment, the memory 18include one or more frame buffers for buffering video data as it isbeing output to the display 28.

In addition to the memory device 18, the electronic device 10 mayfurther include a non-volatile storage 20 for persistent storage of dataand/or instructions. The non-volatile storage 20 may include flashmemory, a hard drive, or any other optical, magnetic, and/or solid-statestorage media, or some combination thereof. Thus, although depicted as asingle device in FIG. 1 for purposes of clarity, it should understoodthat the non-volatile storage device(s) 20 may include a combination ofone or more of the above-listed storage devices operating in conjunctionwith the processor(s) 16. The non-volatile storage 20 may be used tostore firmware, data files, image data, software programs andapplications, wireless connection information, personal information,user preferences, and any other suitable data. In accordance withaspects of the present disclosure, image data stored in the non-volatilestorage 20 and/or the memory device 18 may be processed by the imageprocessing circuitry 32 prior to being output on a display.

The embodiment illustrated in FIG. 1 may also include one or more cardor expansion slots. The card slots may be configured to receive anexpansion card 22 that may be used to add functionality, such asadditional memory, I/O functionality, or networking capability, to theelectronic device 10. Such an expansion card 22 may connect to thedevice through any type of suitable connector, and may be accessedinternally or external with respect to a housing of the electronicdevice 10. For example, in one embodiment, the expansion card 24 may beflash memory card, such as a SecureDigital (SD) card, mini- or microSD,CompactFlash card, or the like, or may be a PCMCIA device. Additionally,the expansion card 24 may be a Subscriber Identity Module (SIM) card,for use with an embodiment of the electronic device 10 that providesmobile phone capability.

The electronic device 10 also includes the network device 24, which maybe a network controller or a network interface card (NIC) that mayprovide for network connectivity over a wireless 802.11 standard or anyother suitable networking standard, such as a local area network (LAN),a wide area network (WAN), such as an Enhanced Data Rates for GSMEvolution (EDGE) network, a 3G data network, or the Internet. In certainembodiments, the network device 24 may provide for a connection to anonline digital media content provider, such as the iTunes® musicservice, available from Apple Inc.

The power source 26 of the device 10 may include the capability to powerthe device 10 in both non-portable and portable settings. For example,in a portable setting, the device 10 may include one or more batteries,such as a Li-Ion battery, for powering the device 10. The battery may bere-charged by connecting the device 10 to an external power source, suchas to an electrical wall outlet. In a non-portable setting, the powersource 26 may include a power supply unit (PSU) configured to draw powerfrom an electrical wall outlet, and to distribute the power to variouscomponents of a non-portable electronic device, such as a desktopcomputing system.

The display 28 may be used to display various images generated by device10, such as a GUI for an operating system, or image data (includingstill images and video data) processed by the image processing circuitry32, as will be discussed further below. As mentioned above, the imagedata may include image data acquired using the imaging device 30 orimage data retrieved from the memory 18 and/or non-volatile storage 20.The display 28 may be any suitable type of display, such as a liquidcrystal display (LCD), plasma display, or an organic light emittingdiode (OLED) display, for example. Additionally, as discussed above, thedisplay 28 may be provided in conjunction with the above-discussedtouch-sensitive mechanism (e.g., a touch screen) that may function aspart of a control interface for the electronic device 10.

The illustrated imaging device(s) 30 may be provided as a digital cameraconfigured to acquire both still images and moving images (e.g., video).The camera 30 may include a lens and one or more image sensorsconfigured to capturing and converting light into electrical signals. Byway of example only, the image sensor may include a CMOS image sensor(e.g., a CMOS active-pixel sensor (APS)) or a CCD (charge-coupleddevice) sensor. Generally, the image sensor in the camera 30 includes anintegrated circuit having an array of pixels, wherein each pixelincludes a photodetector for sensing light. As those skilled in the artwill appreciate, the photodetectors in the imaging pixels generallydetect the intensity of light captured via the camera lenses. However,photodetectors, by themselves, are generally unable to detect thewavelength of the captured light and, thus, are unable to determinecolor information.

Accordingly, the image sensor may further include a color filter array(CFA) that may overlay or be disposed over the pixel array of the imagesensor to capture color information. The color filter array may includean array of small color filters, each of which may overlap a respectivepixel of the image sensor and filter the captured light by wavelength.Thus, when used in conjunction, the color filter array and thephotodetectors may provide both wavelength and intensity informationwith regard to light captured through the camera, which may berepresentative of a captured image.

In one embodiment, the color filter array may include a Bayer colorfilter array, which provides a filter pattern that is 50% greenelements, 25% red elements, and 25% blue elements. For instance, FIG. 2shows a 2×2 pixel block of a Bayer CFA includes 2 green elements (Gr andGb), 1 red element (R), and 1 blue element (B). Thus, an image sensorthat utilizes a Bayer color filter array may provide informationregarding the intensity of the light received by the camera 30 at thegreen, red, and blue wavelengths, whereby each image pixel records onlyone of the three colors (RGB). This information, which may be referredto as “raw image data” or data in the “raw domain,” may then beprocessed using one or more demosaicing techniques to convert the rawimage data into a full color image, generally by interpolating a set ofred, green, and blue values for each pixel. As will be discussed furtherbelow, such demosaicing techniques may be performed by the imageprocessing circuitry 32.

As mentioned above, the image processing circuitry 32 may provide forvarious image processing steps, such as defective pixeldetection/correction, lens shading correction, demosaicing, and imagesharpening, noise reduction, gamma correction, image enhancement,color-space conversion, image compression, chroma sub-sampling, andimage scaling operations, and so forth. In some embodiments, the imageprocessing circuitry 32 may include various subcomponents and/ordiscrete units of logic that collectively form an image processing“pipeline” for performing each of the various image processing steps.These subcomponents may be implemented using hardware (e.g., digitalsignal processors or ASICs) or software, or via a combination ofhardware and software components. The various image processingoperations that may be provided by the image processing circuitry 32and, particularly those processing operations relating to defectivepixel detection/correction, lens shading correction, demosaicing, andimage sharpening, will be discussed in greater detail below.

Before continuing, it should be noted that while various embodiments ofthe various image processing techniques discussed below may utilize aBayer CFA, the presently disclosed techniques are not intended to belimited in this regard. Indeed, those skilled in the art will appreciatethat the image processing techniques provided herein may be applicableto any suitable type of color filter array, including RGBW filters, CYGMfilters, and so forth.

Referring again to the electronic device 10, FIGS. 3-6 illustratevarious forms that the electronic device 10 may take. As mentionedabove, the electronic device 10 may take the form of a computer,including computers that are generally portable (such as laptop,notebook, and tablet computers) as well as computers that are generallynon-portable (such as desktop computers, workstations and/or servers),or other type of electronic device, such as handheld portable electronicdevices (e.g., digital media player or mobile phone). In particular,FIGS. 3 and 4 depict the electronic device 10 in the form of a laptopcomputer 40 and a desktop computer 50, respectively. FIGS. 5 and 6 showfront and rear views, respectively, of the electronic device 10 in theform of a handheld portable device 60.

As shown in FIG. 3, the depicted laptop computer 40 includes a housing42, the display 28, the I/O ports 12, and the input structures 14. Theinput structures 14 may include a keyboard and a touchpad mouse that areintegrated with the housing 42. Additionally, the input structure 14 mayinclude various other buttons and/or switches which may be used tointeract with the computer 40, such as to power on or start thecomputer, to operate a GUI or an application running on the computer 40,as well as adjust various other aspects relating to operation of thecomputer 40 (e.g., sound volume, display brightness, etc.). The computer40 may also include various I/O ports 12 that provide for connectivityto additional devices, as discussed above, such as a FireWire® or USBport, a high definition multimedia interface (HDMI) port, or any othertype of port that is suitable for connecting to an external device.Additionally, the computer 40 may include network connectivity (e.g.,network device 26), memory (e.g., memory 20), and storage capabilities(e.g., storage device 22), as described above with respect to FIG. 1.

Further, the laptop computer 40, in the illustrated embodiment, mayinclude an integrated imaging device 30 (e.g., camera). In otherembodiments, the laptop computer 40 may utilize an external camera(e.g., an external USB camera or a “webcam”) connected to one or more ofthe I/O ports 12 instead of or in addition to the integrated camera 30.For instance, an external camera may be an iSight® camera available fromApple Inc. The camera 30, whether integrated or external, may providefor the capture and recording of images. Such images may then be viewedby a user using an image viewing application, or may be utilized byother applications, including video-conferencing applications, such asiChat®, and image editing/viewing applications, such as Photo Booth®,Aperture®, iPhoto®, or Preview®, which are available from Apple Inc. Incertain embodiments, the depicted laptop computer 40 may be a model of aMacBook®, MacBook® Pro, MacBook Air®, or PowerBook® available from AppleInc. As discussed above, the computer 40 may also be a tablet computingdevice, such as a model of an iPad® from Apple Inc.

FIG. 4 further illustrates an embodiment in which the electronic device10 is provided as a desktop computer 50. As will be appreciated, thedesktop computer 50 may include a number of features that may begenerally similar to those provided by the laptop computer 40 shown inFIG. 4, but may have a generally larger overall form factor. As shown,the desktop computer 50 may be housed in an enclosure 42 that includesthe display 28, as well as various other components discussed above withregard to the block diagram shown in FIG. 1. Further, the desktopcomputer 50 may include an external keyboard and mouse (input structures14) that may be coupled to the computer 50 via one or more I/O ports 12(e.g., USB) or may communicate with the computer 50 wirelessly (e.g.,RF, Bluetooth, etc.). The desktop computer 50 also includes an imagingdevice 30, which may be an integrated or external camera, as discussedabove. In certain embodiments, the depicted desktop computer 50 may be amodel of an iMac®, Mac® mini, or Mac Pro®, available from Apple Inc.

As further shown, the display 28 may be configured to generate variousimages that may be viewed by a user. For example, during operation ofthe computer 50, the display 28 may display a graphical user interface(“GUI”) 52 that allows the user to interact with an operating systemand/or application running on the computer 50. The GUI 52 may includevarious layers, windows, screens, templates, or other graphical elementsthat may be displayed in all, or a portion, of the display device 28.For instance, in the depicted embodiment, an operating system GUI 52 mayinclude various graphical icons 54, each of which may correspond tovarious applications that may be opened or executed upon detecting auser selection (e.g., via keyboard/mouse or touchscreen input). Theicons 54 may be displayed in a dock 56 or within one or more graphicalwindow elements 58 displayed on the screen. In some embodiments, theselection of an icon 54 may lead to a hierarchical navigation process,such that selection of an icon 54 leads to a screen or opens anothergraphical window that includes one or more additional icons or other GUIelements. By way of example only, the operating system GUI 52 displayedin FIG. 4 may be from a version of the Mac OS® operating system,available from Apple Inc.

Continuing to FIGS. 5 and 6, the electronic device 10 is furtherillustrated in the form of portable handheld electronic device 60, whichmay be a model of an iPod® or iPhone® available from Apple Inc. In thedepicted embodiment, the handheld device 60 includes an enclosure 42,which may function to protect the interior components from physicaldamage and to shield them from electromagnetic interference. Theenclosure 42 may be formed from any suitable material or combination ofmaterials, such as plastic, metal, or a composite material, and mayallow certain frequencies of electromagnetic radiation, such as wirelessnetworking signals, to pass through to wireless communication circuitry(e.g., network device 24), which may be disposed within the enclosure42, as shown in FIG. 5.

The enclosure 42 also includes various user input structures 14 throughwhich a user may interface with the handheld device 60. For instance,each input structure 14 may be configured to control one or morerespective device functions when pressed or actuated. By way of example,one or more of the input structures 14 may be configured to invoke a“home” screen 42 or menu to be displayed, to toggle between a sleep,wake, or powered on/off mode, to silence a ringer for a cellular phoneapplication, to increase or decrease a volume output, and so forth. Itshould be understood that the illustrated input structures 14 are merelyexemplary, and that the handheld device 60 may include any number ofsuitable user input structures existing in various forms includingbuttons, switches, keys, knobs, scroll wheels, and so forth.

As shown in FIG. 5, the handheld device 60 may include various I/O ports12. For instance, the depicted I/O ports 12 may include a proprietaryconnection port 12 a for transmitting and receiving data files or forcharging a power source 26 and an audio connection port 12 b forconnecting the device 60 to an audio output device (e.g., headphones orspeakers). Further, in embodiments where the handheld device 60 providesmobile phone functionality, the device 60 may include an I/O port 12 cfor receiving a subscriber identify module (SIM) card (e.g., anexpansion card 22).

The display device 28, which may be an LCD, OLED, or any suitable typeof display, may display various images generated by the handheld device60. For example, the display 28 may display various system indicators 64providing feedback to a user with regard to one or more states ofhandheld device 60, such as power status, signal strength, externaldevice connections, and so forth. The display may also display a GUI 52that allows a user to interact with the device 60, as discussed abovewith reference to FIG. 4. The GUI 52 may include graphical elements,such as the icons 54 which may correspond to various applications thatmay be opened or executed upon detecting a user selection of arespective icon 54. By way of example, one of the icons 54 may representa camera application 66 that may be used in conjunction with a camera 30(shown in phantom lines in FIG. 5) for acquiring images. Referringbriefly to FIG. 6, a rear view of the handheld electronic device 60depicted in FIG. 5 is illustrated, which shows the camera 30 as beingintegrated with the housing 42 and positioned on the rear of thehandheld device 60.

As mentioned above, image data acquired using the camera 30 may beprocessed using the image processing circuitry 32, which my includehardware (e.g., disposed within the enclosure 42) and/or software storedon one or more storage devices (e.g., memory 18 or non-volatile storage20) of the device 60. Images acquired using the camera application 66and the camera 30 may be stored on the device 60 (e.g., in storagedevice 20) and may be viewed at a later time using a photo viewingapplication 68.

The handheld device 60 may also include various audio input and outputelements. For example, the audio input/output elements, depictedgenerally by reference numeral 70, may include an input receiver, suchas one or more microphones. For instance, where the handheld device 60includes cell phone functionality, the input receivers may be configuredto receive user audio input, such as a user's voice. Additionally, theaudio input/output elements 70 may include one or more outputtransmitters. Such output transmitters may include one or more speakerswhich may function to transmit audio signals to a user, such as duringthe playback of music data using a media player application 72. Further,in embodiments where the handheld device 60 includes a cell phoneapplication, an additional audio output transmitter 74 may be provided,as shown in FIG. 5. Like the output transmitters of the audioinput/output elements 70, the output transmitter 74 may also include oneor more speakers configured to transmit audio signals to a user, such asvoice data received during a telephone call. Thus, the audioinput/output elements 70 and 74 may operate in conjunction to functionas the audio receiving and transmitting elements of a telephone.

Having now provided some context with regard to various forms that theelectronic device 10 may take, the present discussion will now focus onthe image processing circuitry 32 depicted in FIG. 1. As mentionedabove, the image processing circuitry 32 may be implemented usinghardware and/or software components, and may include various processingunits that define an image signal processing (ISP) pipeline. Inparticular, the following discussion may focus on aspects of the imageprocessing techniques set forth in the present disclosure, particularlythose relating to defective pixel detection/correction techniques, lensshading correction techniques, demosaicing techniques, and imagesharpening techniques.

Referring now to FIG. 7, a simplified top-level block diagram depictingseveral functional components that may be implemented as part of theimage processing circuitry 32 is illustrated, in accordance with oneembodiment of the presently disclosed techniques. Particularly, FIG. 7is intended to illustrate how image data may flow through the imageprocessing circuitry 32, in accordance with at least one embodiment. Inorder to provide a general overview of the image processing circuitry32, a general description of how these functional components operate toprocess image data is provided here with reference to FIG. 7, while amore specific description of each of the illustrated functionalcomponents, as well as their respective sub-components, will be furtherprovided below.

Referring to the illustrated embodiment, the image processing circuitry32 may include image signal processing (ISP) front-end processing logic80, ISP pipe processing logic 82, and control logic 84. Image datacaptured by the imaging device 30 may first be processed by the ISPfront-end logic 80 and analyzed to capture image statistics that may beused to determine one or more control parameters for the ISP pipe logic82 and/or the imaging device 30. The ISP front-end logic 80 may beconfigured to capture image data from an image sensor input signal. Forinstance, as shown in FIG. 7, the imaging device 30 may include a camerahaving one or more lenses 88 and image sensor(s) 90. As discussed above,the image sensor(s) 90 may include a color filter array (e.g., a Bayerfilter) and may thus provide both light intensity and wavelengthinformation captured by each imaging pixel of the image sensors 90 toprovide for a set of raw image data that may be processed by the ISPfront-end logic 80. For instance, the output 92 from the imaging device30 may be received by a sensor interface 94, which may then provide theraw image data 96 to the ISP front-end logic 80 based, for example, onthe sensor interface type. By way of example, the sensor interface 94may utilize a Standard Mobile Imaging Architecture (SMIA) interface or aMobile Industry Processor Interface (MIPI), or some combination thereof.In certain embodiments, the ISP front-end logic 80 may operate withinits own clock domain and may provide an asynchronous interface to thesensor interface 94 to support image sensors of different sizes andtiming requirements.

The raw image data 96 may be provided to the ISP front-end logic 80 andprocessed on a pixel-by-pixel basis in a number of formats. Forinstance, each image pixel may have a bit-depth of 8, 10, 12, or 14bits. The ISP front-end logic 80 may perform one or more imageprocessing operations on the raw image data 96, as well as collectstatistics about the image data 96. The image processing operations, aswell as the collection of statistical data, may be performed at the sameor at different bit-depth precisions. For example, in one embodiment,processing of the raw image pixel data 96 may be performed at aprecision of 14-bits. In such embodiments, raw pixel data received bythe ISP front-end logic 80 that has a bit-depth of less than 14 bits(e.g., 8-bit, 10-bit, 12-bit) may be up-sampled to 14-bits for imageprocessing purposes. In another embodiment, statistical processing mayoccur at a precision of 8-bits and, thus, raw pixel data having a higherbit-depth may be down-sampled to an 8-bit format for statisticspurposes. As will be appreciated, down-sampling to 8-bits may reducehardware size (e.g., area) and also reduce processing/computationalcomplexity for the statistics data. Additionally, the raw image data maybe averaged spatially to allow for the statistics data to be more robustto noise.

Further, as shown in FIG. 7, the ISP front-end logic 80 may also receivepixel data from the memory 108. For instance, as shown by referencenumber 98, the raw pixel data may be sent to the memory 108 from thesensor interface 94. The raw pixel data residing in the memory 108 maythen be provided to the ISP front-end logic 80 for processing, asindicated by reference number 100. The memory 108 may be part of thememory device 18, the storage device 20, or may be a separate dedicatedmemory within the electronic device 10 and may include direct memoryaccess (DMA) features. Further, in certain embodiments, the ISPfront-end logic 80 may operate within its own clock domain and providean asynchronous interface to the sensor interface 94 to support sensorsof different sizes and having different timing requirements.

Upon receiving the raw image data 96 (from sensor interface 94) or 100(from memory 108), the ISP front-end logic 80 may perform one or moreimage processing operations, such as temporal filtering and/or binningcompensation filtering. The processed image data may then be provided tothe ISP pipe logic 82 (output signal 109) for additional processingprior to being displayed (e.g., on display device 28), or may be sent tothe memory (output signal 110). The ISP pipe logic 82 receives the“front-end” processed data, either directly form the ISP front-end logic80 or from the memory 108 (input signal 112), and may provide foradditional processing of the image data in the raw domain, as well as inthe RGB and YCbCr color spaces. Image data processed by the ISP pipelogic 82 may then be output (signal 114) to the display 28 for viewingby a user and/or may be further processed by a graphics engine or GPU.Additionally, output from the ISP pipe logic 82 may be sent to memory108 (signal 115) and the display 28 may read the image data from memory108 (signal 116), which may, in certain embodiments, be configured toimplement one or more frame buffers. Further, in some implementations,the output of the ISP pipe logic 82 may also be provided to acompression/decompression engine 118 (signal 117) for encoding/decodingthe image data. The encoded image data may be stored and the laterdecompressed prior to being displayed on the display 28 (signal 119). Byway of example, the compression engine or “encoder” 118 may be a JPEGcompression engine for encoding still images, or an H.264 compressionengine for encoding video images, or some combination thereof, as wellas a corresponding decompression engine for decoding the image data.Additional information with regard to image processing operations thatmay be provided in the ISP pipe logic 82 will be discussed in greaterdetail below with regard to FIGS. 27-57. Also, it should be noted thatthe ISP pipe logic 82 may also receive raw image data from the memory108, as depicted by input signal 112.

Statistical data 102 determined by the ISP front-end logic 80 may beprovided to a control logic unit 84. The statistical data 102 mayinclude, for example, image sensor statistics relating to auto-exposure,auto-white balance, auto-focus, flicker detection, black levelcompensation (BLC), lens shading correction, and so forth. The controllogic 84 may include a processor and/or microcontroller configured toexecute one or more routines (e.g., firmware) that may be configured todetermine, based upon the received statistical data 102, controlparameters 104 for the imaging device 30, as well as control parameters106 for the ISP pipe processing logic 82. By way of example only, thecontrol parameters 104 may include sensor control parameters (e.g.,gains, integration time for exposure control), camera flash controlparameters, lens control parameters (e.g., focal length for focusing orzoom), or a combination of such parameters. The ISP control parameters106 may include gain levels and color correction matrix (CCM)coefficients for auto-white balance and color adjustment (e.g., duringRGB processing), as well as lens shading correction parameters which, asdiscussed below, may be determined based upon white point balanceparameters. In some embodiments, the control logic 84 may, in additionto analyzing statistics data 102, also analyze historical statistics,which may be stored on the electronic device 10 (e.g., in memory 18 orstorage 20).

Due to the generally complex design of the image processing circuitry 32shown herein, it may be beneficial to separate the discussion of the ISPfront-end logic 80 and the ISP pipe processing logic 82 into separatesections, as shown below. Particularly, FIGS. 8 to 26 of the presentapplication may relate to the discussion of various embodiments andaspects of the ISP front-end logic 80, while FIGS. 27 to 57 of thepresent application may relate to the discussion of various embodimentsand aspects of the ISP pipe processing logic 82.

The ISP Front-End Processing Logic

FIG. 8 is a more detailed block diagram showing functional logic blocksthat may be implemented in the ISP front-end logic 80, in accordancewith one embodiment. Depending on the configuration of the imagingdevice 30 and/or sensor interface 94, as discussed above in FIG. 7, rawimage data may be provided to the ISP front-end logic 80 by one or moreimage sensors 90. In the depicted embodiment, raw image data may beprovided to the ISP front-end logic 80 by a first image sensor 90 a(Sensor0) and a second image sensor 90 b (Sensor1). As shown, the imagesensors 90 a and 90 b may provide the raw image data as signals Sif0 andSif1, respectively. Each of the image sensors 90 a and 90 b may beassociated with the respective statistics processing units 120 and 122.

Further, it should be noted that the raw image data Sif0 and Sif1 may beprovided directly to their respective processing units 120 and 122, ormay be stored in or written to the memory 108 and subsequently retrievedas signals SifIn0 and SifIn1, respectively. Thus, the ISP front-endlogic 80 may include selection logic 124, which may provide either theSif0 or SifIn0 signals representative of raw image data captured bySensor0 (90 a) to the statistics processing unit 120, and may alsoinclude the selection logic 126, which may provide either the Sif1 orSifIn1 signals representative of raw image data captured by Sensor1 (90b) to the statistics processing unit 122. Each of the statisticsprocessing units 120 and 122 may determine various statistical dataobtained via analysis of the raw image sensor data, and may respectivesets of statistics, as indicated by the output signals Stats0 andStats1. As mentioned above, the statistical data (Stats0 and Stats1) maybe provided to the control logic 84 for the determination of variouscontrol parameters that may be used to operate the imaging device 30and/or the ISP pipe processing logic 82.

In addition to the statistics processing units 120 and 122, the ISPfront-end logic 80 may further include a pixel processing unit 130. Thepixel processing unit 130 may perform various image processingoperations on the raw image data on a pixel-by-pixel basis. As shown,the pixel processing unit 130, by way of the selection logic 132, mayreceive the raw image signals Sif0 (from Sensor0) or Sif1 (fromSensor1), or may receive raw image data FEProcIn from the memory 108. Ascan be appreciated, the selection logic blocks (120, 122, and 132) shownin FIG. 8 may be provided by any suitable type of logic, such as amultiplexer that selects one of multiple input signals in response to acontrol signal. The pixel processing unit 130 may also receive andoutput various signals (e.g., Rin, Hin, Hout, and Yout—which mayrepresent motion history and luma data used during temporal filtering)when performing the pixel processing operations, as will be discussedfurther below. The output 109 (FEProcOut) of the pixel processing unit130 may then be forwarded to the ISP pipe logic 82, such as via one ormore first-in-first-out (FIFO) queues, or may be sent to the memory 108.

Before continuing with a more detailed description of the statisticsprocessing and pixel processing operations depicted in the ISP front-endlogic 80 of FIG. 8, it is believed that a brief introduction regardingthe definitions of various ISP frame regions will help to facilitate abetter understanding of the present subject matter. With this in mind,various frame regions that may be defined within an image source frameare illustrated in FIG. 9. The format for a source frame provided to theimage processing circuitry 32 may use either the tiled or linearaddressing modes discussed above, as may utilize pixel formats in 8, 10,12, or 14-bit precision. The image source frame 150, as shown in FIG. 9,may include a sensor frame region 152, a raw frame region 152, and anactive region 154. The sensor frame 152 is generally the maximum framesize that the image sensor 90 can provide to the image processingcircuitry 32. The raw frame region 154 may be defined as the region ofthe sensor frame 152 that is sent to the ISP front-end processing logic80. The active region 156 may be defined as a portion of the sourceframe 150, typically within the raw frame region 154, on whichprocessing is performed for a particular image processing operation. Inaccordance with embodiments of the present technique, that active region156 may be the same or may be different for different image processingoperations.

In accordance with aspects of the present technique, the ISP front-endlogic 80 only receives the raw frame 154. Thus, for the purposes of thepresent discussion, the global frame size for the ISP front-endprocessing logic 80 may be assumed as the raw frame size, as determinedby the width 158 and height 160. In some embodiments, the offset fromthe boundaries of the sensor frame 152 to the raw frame 154 may bedetermined and/or maintained by the control logic 84. For instance, thecontrol logic 84 may be include firmware that may determine the rawframe region 154 based upon input parameters, such as the x-offset 162and the y-offset 164, that are specified relative to the sensor frame152. Further, in some cases, a processing unit within the ISP front-endlogic 80 or the ISP pipe logic 82 may have a defined active region, suchthat pixels in the raw frame but outside the active region 156 will notbe processed, i.e., left unchanged. For instance, an active region 156for a particular processing unit having a width 166 and height 168 maybe defined based upon an x-offset 170 and y-offset 172 relative to theraw frame 154. Further, where an active region is not specificallydefined, one embodiment of the image processing circuitry 32 may assumethat the active region 156 is the same as the raw frame 154 (e.g.,x-offset 170 and y-offset 172 are both equal to 0). Thus, for thepurposes of image processing operations performed on the image data,boundary conditions may be defined with respect to the boundaries of theraw frame 154 or active region 156.

Keeping these points in mind and referring to FIG. 10, a more detailedview of the ISP front-end pixel processing logic 130 (previouslydiscussed in FIG. 8) is illustrated, in accordance with an embodiment ofthe present technique. As shown, the ISP front-end pixel processinglogic 130 includes a temporal filter 180 and a binning compensationfilter 182. The temporal filter 180 may receive one of the input imagesignals Sif0, Sif1, or FEProcIn, and may operate on the raw pixel databefore any additional processing is performed. For example, the temporalfilter 180 may initially process the image data to reduce noise byaveraging image frames in the temporal direction.

The temporal filter 180 may be pixel-adaptive based upon motion andbrightness characteristics. For instance, when pixel motion is high, thefiltering strength may be reduced in order to avoid the appearance of“trailing” or “ghosting artifacts” in the resulting processed image,whereas the filtering strength may be increased when little or no motionis detected. Additionally, the filtering strength may also be adjustedbased upon brightness data (e.g., “luma”). For instance, as imagebrightness increases, filtering artifacts may become more noticeable tothe human eye. Thus, the filtering strength may be further reduced whena pixel has a high level of brightness.

In applying temporal filtering, the temporal filter 180 may receivereference pixel data (Rin) and motion history input data (Hin), whichmay be from a previous filtered or original frame. Using theseparameters, the temporal filter 180 may provide motion history outputdata (Hout) and filtered pixel output (Yout). The filtered pixel outputYout is then passed to the binning compensation filter 182, which may beconfigured to perform one or more scaling operations on the filteredpixel output data Yout to produce the output signal FEProcOut. Theprocessed pixel data FEProcOut may then be forwarded to the ISP pipeprocessing logic 82, as discussed above.

Referring to FIG. 11, a process diagram depicting a temporal filteringprocess 190 that may be performed by the temporal filter shown in FIG.10 is illustrated, in accordance with one embodiment. The temporalfilter 180 may include a 2-tap filter, wherein the filter coefficientsare adjusted adaptively on a per pixel basis based at least partiallyupon motion and brightness data. For instance, input pixels x(t), withthe variable “t” denoting a temporal value, may be compared to referencepixels r(t−1) in a previously filtered frame or a previous originalframe to generate a motion index lookup in a motion history table (M)192 that may contain filter coefficients. Additionally, based uponmotion history input data h(t−1), a motion history output h(t)corresponding to the current input pixel x(t) may be determined.

The motion history output h(t) and a filter coefficient, K, may bedetermined based upon a motion delta d(j,i,t), wherein (j,i) representcoordinates of the spatial location of a current pixel x(j,i,t). Themotion delta d(j,i,t) may be computed by determining the maximum ofthree absolute deltas between original and reference pixels for threehorizontally collocated pixels of the same color. For instance,referring briefly to FIG. 12, the spatial locations of three collocatedreference pixels 200, 202, and 204 that corresponding to original inputpixels 206, 208, and 210 are illustrated. In one embodiment, the motiondelta may be calculated based on these original and reference pixelsusing formula below:d(j,i,t)=max 3[abs(x(j,i−2,t)−r(j,i−2,t)),(abs(x(j,i,t)−r(j,i,t)),(abs(x(j,i+2,t)−r(j,i+2,t))]  (1)

Referring back to FIG. 11, once the motion delta value is determined, amotion index lookup that may be used to selected the filter coefficientK from the motion table (M) 192 may be calculated by summing the motiondelta d(t) for the current pixel (e.g., at spatial location (j,i)) withthe motion history input h(t−1). For instance, the filter coefficient Kmay be determined as follows:K=M[d(j,i,t)+h(j,i,t−1)]  (2)Additionally, the motion history output h(t) may be determined using thefollowing formula:h(j,i,t)=d(j,i,t)+(1−K)×h(j,i,t−1)  (3)

Next, the brightness of the current input pixel x(t) may be used togenerate a luma index lookup in a luma table (L) 194. In one embodiment,the luma table may contain attenuation factors that may be between 0 and1, and may be selected based upon the luma index. A second filtercoefficient, K′, may be calculated by multiplying the first filtercoefficient K by the luma attenuation factor, as shown in the followingequation:K′=K×L[x(j,i,t)]  (4)

The determined value for K′ may then be used as the filteringcoefficient for the temporal filter 180. As discussed above, thetemporal filter 180 may be a 2-tap filter. Additionally, the temporalfilter 180 may be configured as an infinite impulse response (IIR)filter using previous filtered frame or as a finite impulse response(FIR) filter using previous original frame. The temporal filter 180 maycompute the filtered output pixel y(t) (Yout) using the current inputpixel x(t), the reference pixel r(t−1), and the filter coefficient K′using the following formula:y(j,i,t)=r(j,i,t−1)+K′(x(j,i,t)−r(j,i,t−1))  (5)As discussed above, the temporal filtering process 190 shown in FIG. 11may be performed on a pixel-by-pixel basis. In one embodiment, the samemotion table M and luma table L may be used for all color components(e.g., R, G, and B). Additionally, some embodiments may provide a bypassmechanism, in which temporal filtering may be bypassed, such as inresponse to a control signal from the control logic 84

The output of the temporal filter 180 may subsequently be sent to thebinning compensation filter (BCF) 182, which may process the imagepixels to compensate for non-linear placement of the color samples, suchthat subsequent image processing operations in the ISP pipe logic 82(e.g., demosaicing, etc.) that depend on linear placement of the colorsamples can operate correctly. Additionally, the BCF 182 may alsoprocess the pixel data by applying one or more scaling operations, suchas vertical and/or horizontal scaling. FIG. 13 shows a block diagram ofthe binning compensation filter 182 that may include scaling logic 214and a differential analyzer 216, in accordance with one disclosedembodiment, and FIG. 14 shows a process 220 that may be utilized forcarrying out the scaling operations.

As shown in FIG. 13, the scaling logic 214 may produce the outputFEProcOut (109) which, as discussed above, may be forwarded to the ISPpipe logic 82 for additional processing, as will be discussed furtherbelow. In one embodiment, the scaling operation(s) performed by the BCF182 may be performed using one or more multi-tap polyphase filters whichmay select pixels from the source image, multiply each pixel by aweighting factor, and then summing up the pixel values to form adestination pixel. As will be appreciated, the selection of the pixelsused in the scaling operations may depend at least partially upon thecurrent pixel position and the number of taps provided by the filters.Additionally, the filtering operations may be done per color componentusing same colored pixels, and weighting factors (or coefficients) maybe obtained form a lookup table and determined based upon the currentbetween-pixel fractional position.

In the depicted embodiment, the differential analyzer 216 may be adigital differential analyzer (DDA) and may be configured to control thecurrent pixel position during the scaling operations. By way of exampleonly, the DDA 216 may be provided as a 32-bit data register thatcontains a two's-complement fixed-point number having 12 bits in theinteger portion and 20 bits in the fraction. The 12-bit integer portionmay be used to determine the current pixel position. The fractionalportion is used as an index into a coefficient table. In one embodiment,the vertical and horizontal scaling components may utilize 8-deepcoefficient tables, such that the high-order 3 bits of the 20-bitfraction portion are used for the index.

To provide an example based upon a 32-bit DDA register (DDA[31:0]), acurrent center source pixel location (currPixel) may be defined by the12-bit integer portion DDA[31:20], and may be rounded up (+1) if thenext bit DDA[19] is 1. Source pixel values of pixels neighboringcurrPixel may then be obtained depending on the number of taps providedby the filter. For instance, in one embodiment, vertical scaling may beperformed using a 3-tap polyphase filter, such that one pixel of thesame color on each side of currPixel is selected (e.g., −1, 0, +1), andhorizontal scaling may be performed using a 5-tap polyphase filter,wherein two pixels of the same color on each side of currPixel areselected (e.g., −2, −1, 0, +1, +2). Further, each tap may have its ownrespective coefficient table. Thus, three 8-deep tables may be providedfor a 3-tap vertical scaling filter, and five 8-deep tables may beprovided for a 5-tap horizontal scaling filter. The current coefficientindex (currIndex) may be define by DDA[19:16] (the high-order 3 bits ofthe fraction portion DDA[19:0]), and may rounded up (+1) if the nextbit, DDA[15] is 1.

Accordingly, the filtering process that occurs in the BCF 182 mayinclude obtaining the source pixel values around the center pixel(currPixel) and multiplying them by the appropriate coefficients fromthe tables accessed using currIndex. Once the filtering process iscompleted for a given pixel, a step value (DDAStep) may be added to theDDA 216 to determine the position of the next pixel, and thefiltering/scaling operations may be repeated for the subsequent pixel.This is further shown by the process 220 illustrated in FIG. 14. Forinstance, beginning at step 222, the DDA 216 is initialized and acurrent pixel position is identified. As discussed above, where the DDA216 includes a 32-bit register for storing a two's complementfixed-point number, the current pixel position may be specified by theupper 12-bits (integer portion) of the register data.

At step 224, multi-tap filtering is performed for both vertical andhorizontal scaling. By way of example, assuming that 3-tap and 5-tappolyphase filters are used for vertical and horizontal scaling,respectively, and assuming that the image sensor 90 uses a Bayer colorfilter pattern (FIG. 2), the vertical scaling component may include fourseparate 3-tap polyphase filters, one for each color component: Gr, R,B, and Gb. Each of the 3-tap filters may use a DDA to control thestepping of the current center pixel and the index for the coefficients.Similarly, the horizontal scaling components may include four separate5-tap polyphase filters, one for each color component: Gr, R, B, and Gb.Each of the 5-tap filters may use a DDA to control the stepping of thecurrent center pixel and the index for the coefficients. For boundarycases, the pixels used in the horizontal and vertical filtering processmay depend upon the position of the current pixel, as established by theDDA. For instance, if the DDA values indicate the current pixel(currPixel) is close to a frame border, such that one or more of thesource pixels needed by the filter lie outside of the border, the borderpixels may be repeated.

Finally, as shown at step 226, once the vertical and horizontal scalingoperations have been completed for a given current pixel (currPixel),the DDA 216 may be stepped to the position of the next pixel, and theprocess 220 may return to step 224 to perform vertical and horizontalscaling on the next pixel. As discussed above the output of the BCF 182,which may be the output FEProcOut (109), may be forwarded to the ISPpipe processing logic 82 for additional processing. However, beforeshifting the focus of this discussion to the ISP pipe processing logic82, a more detailed description of various functionalities that may beprovided by the statistics processing units (e.g., 122 and 124) that maybe implemented in the ISP front-end logic 80 will first be provided.

Referring back to the general description of the statistics processingunits 120 and 122, these units may be configured to collect variousstatistics about the image sensors that capture and provide the rawimage signals (Sif0 and Sif1), such as statistics relating toauto-exposure, auto-white balance, auto-focus, flicker detection, blacklevel compensation, and lens shading correction, and so forth. In doingso, the statistics processing units 120 and 122 may first apply one ormore image processing operations to their respective input signals, Sif0(from Sensor0) and Sif1 (from Sensor1).

For example, referring to FIG. 15, a more detailed block diagram view ofthe statistics processing unit 120 associated with Sensor0 (90 a) isillustrated in accordance with one embodiment. As shown, the statisticsprocessing unit 120 may include the following functional blocks:defective pixel detection and correction logic 230, black levelcompensation (BLC) logic 232, lens shading correction logic 234, inverseBLC logic 236, and statistics collection logic 238. Each of thesefunctional blocks will be discussed below. Further, it should beunderstood that the statistics processing unit 122 associated withSensor1 (90 b) may be implemented in a similar manner.

Initially, the output of selection logic 124 (e.g., Sif0 or SifIn0) isreceived by the front-end defective pixel correction logic 230. As willbe appreciated, “defective pixels” may be understood to refer to imagingpixels within the image sensor(s) 90 that fail to sense light levelsaccurately. Defective pixels may attributable to a number of factors,and may include “hot” (or leaky) pixels, “stuck” pixels, and “deadpixels.” A “hot” pixel generally appears as being brighter than anon-defective pixel given the same amount of light at the same spatiallocation. Hot pixels may result due to reset failures and/or highleakage. For example, a hot pixel may exhibit a higher than normalcharge leakage relative to non-defective pixels, and thus may appearbrighter than non-defective pixels. Additionally, “dead” and “stuck”pixels may be the result of impurities, such as dust or other tracematerials, contaminating the image sensor during the fabrication and/orassembly process, which may cause certain defective pixels to be darkeror brighter than a non-defective pixel, or may cause a defective pixelto be fixed at a particular value regardless of the amount of light towhich it is actually exposed. Additionally, dead and stuck pixels mayalso result from circuit failures that occur during operation of theimage sensor. By way of example, a stuck pixel may appear as alwaysbeing on (e.g., fully charged) and thus appears brighter, whereas a deadpixel appears as always being off.

The defective pixel detection and correction (DPDC) logic 230 in the ISPfront-end logic 80 may correct (e.g., replace defective pixel values)defective pixels before they are considered in statistics collection(e.g., 238). In one embodiment, defective pixel correction is performedindependently for each color component (e.g., R, B, Gr, and Gb for aBayer pattern). Generally, the front-end DPDC logic 230 may provide fordynamic defect correction, wherein the locations of defective pixels aredetermined automatically based upon directional gradients computed usingneighboring pixels of the same color. As will be understand, the defectsmay be “dynamic” in the sense that the characterization of a pixel asbeing defective at a given time may depend on the image data in theneighboring pixels. By way of example, a stuck pixel that is always onmaximum brightness may not be regarded as a defective pixel if thelocation of the stuck pixel is in an area of the current image that isdominate by brighter or white colors. Conversely, if the stuck pixel isin a region of the current image that is dominated by black or darkercolors, then the stuck pixel may be identified as a defective pixelduring processing by the DPDC logic 230 and corrected accordingly.

The DPDC logic 230 may utilize one or more horizontal neighboring pixelsof the same color on each side of a current pixel to determine if thecurrent pixel is defective using pixel-to-pixel directional gradients.If a current pixel is identified as being defective, the value of thedefective pixel may be replaced with the value of a horizontalneighboring pixel. For instance, in one embodiment, five horizontalneighboring pixels of the same color that are inside the raw frame 154(FIG. 9) boundary are used, wherein the five horizontal neighboringpixels include the current pixel and two neighboring pixels on eitherside. Thus, as illustrated in FIG. 16, for a given color component c andfor the current pixel P, horizontal neighbor pixels P0, P1, P2, and P3may be considered by the DPDC logic 230. It should be noted, however,that depending on the location of the current pixel P, pixels outsidethe raw frame 154 are not considered when calculating pixel-to-pixelgradients.

For instance, as shown in FIG. 16, in a “left edge” case 240, thecurrent pixel P is at the leftmost edge of the raw frame 154 and, thus,the neighboring pixels P0 and P1 outside of the raw frame 154 are notconsidered, leaving only the pixels P, P2, and P3 (N=3). In a “leftedge+1” case 242, the current pixel P is one unit pixel away from theleftmost edge of the raw frame 154 and, thus, the pixel P0 is notconsidered. This leaves only the pixels P1, P, P2, and P3 (N=4).Further, in a “centered” case 244, pixels P0 and P1 on the left side ofthe current pixel P and pixels P2 and P3 on the right side of thecurrent pixel P are within the raw frame 154 boundary and, therefore,all of the neighboring pixels P0, P1, P2, and P3 (N=5) are considered incalculating pixel-to-pixel gradients. Additionally, similar cases 246and 248 may be encountered as the rightmost edge of the raw frame 154 isapproached. For instance, given the “right edge−1” case 246, the currentpixel P is one unit pixel away the rightmost edge of the raw frame 154and, thus, the pixel P3 is not considered (N=4). Similarly, in the“right edge” case 248, the current pixel P is at the rightmost edge ofthe raw frame 154 and, thus, both of the neighboring pixels P2 and P3are not considered (N=3).

In the illustrated embodiment, for each neighboring pixel (k=0 to 3)within the picture boundary (e.g., raw frame 154), the pixel-to-pixelgradients may be calculated as follows:G _(k) =abs(P−P _(k)), for 0≦k≦3 (only for k within the raw frame)  (6)Once the pixel-to-pixel gradients have been determined, defective pixeldetection may be performed by the DPDC logic 230 as follows. First, itis assumed that a pixel is defective if a certain number of itsgradients G_(k) are at or below a particular threshold, denoted by thevariable dprTh. Thus, for each pixel, a count (C) of the number ofgradients for neighboring pixels inside the picture boundaries that areat or below the threshold dprTh is accumulated. By way of example, foreach neighbor pixel inside the raw frame 154, the accumulated count C ofthe gradients G_(k) that are at or below the threshold dprTh may becomputed as follows:

$\begin{matrix}{{C = {\overset{N}{\sum\limits_{k}}\left( {G_{k} \leq {dprTh}} \right)}},} & (7)\end{matrix}$

for 0≦k≦3 (only for k within the raw frame)

As will be appreciated, depending on the color components, the thresholdvalue dprTh may vary. Next, if the accumulated count C is determined tobe less than or equal to a maximum count, denoted by the variabledprMaxC, then the pixel may be considered defective. This logic isexpressed below:if (C≦dprMaxC), then the pixel is defective.  (8)

Defective pixels are replaced using a number of replacement conventions.For instance, in one embodiment, a defective pixel may be replaced withthe pixel to its immediate left, P1. At a boundary condition (e.g., P1is outside of the raw frame 154), a defective pixel may replaced withthe pixel to its immediate right, P2. Further, it should be understoodthat replacement values may be retained or propagated for successivedefective pixel detection operations. For instance, referring to the setof horizontal pixels shown in FIG. 16, if P0 or P1 were previouslyidentified by the DPDC logic 230 as being defective pixels, theircorresponding replacement values may be used for the defective pixeldetection and replacement of the current pixel P.

To summarize the above-discussed defective pixel detection andcorrection techniques, a flow chart depicting such a process is providedin FIG. 17 and referred to by reference number 250. As shown, process250 begins at step 252, at which a current pixel (P) is received and aset of neighbor pixels is identified. In accordance with the embodimentdescribed above, the neighbor pixels may include two horizontal pixelsof the same color component from opposite sides of the current pixel(e.g., P0, P1, P2, and P3). Next, at step 254, horizontal pixel-to-pixelgradients are calculated with respect to each neighboring pixel withinthe raw frame 154, as described in Equation 6 above. Thereafter, at step256, a count C of the number of gradients that are less than or equal toa particular threshold dprTh is determined. As shown at decision logic258, if C is less than or equal to dprMaxC, then the process 250continues to step 260, and the current pixel is identified as beingdefective. The defective pixel is then corrected at step 262 using areplacement value. Additionally, referring back to decision logic 258,if C is greater than dprMaxC, then the process continues to step 264,and the current pixel is identified as not being defective, and itsvalue is not changed.

It should be noted that the defective pixel detection/correctiontechniques applied during the ISP front-end statistics processing may beless robust than defective pixel detection/correction that is performedin the ISP pipe logic 82. For instance, as will be discussed in furtherdetail below, defective pixel detection/correction performed in the ISPpipe logic 82 may, in addition to dynamic defect correction, furtherprovide for fixed defect correction, wherein the locations of defectivepixels are known a priori and loaded in one or more defect tables.Further, dynamic defect correction may in the ISP pipe logic 82 may alsoconsider pixel gradients in both horizontal and vertical directions, andmay also provide for the detection/correction of speckling, as will bediscussed below.

Returning to FIG. 15, the output of the DPDC logic 230 is then passed tothe black level compensation (BLC) logic 232. The BLC logic 232 mayprovide for digital gain, offset, and clipping independently for eachcolor component “c” (e.g., R, B, Gr, and Gb for Bayer) on the pixelsused for statistics collection. For instance, as expressed by thefollowing operation, the input value for the current pixel is firstoffset by a signed value, and then multiplied by a gain.Y=(X+O[c])×G[c],  (9)wherein X represents the input pixel value for a given color component c(e.g., R, B, Gr, or Gb), O[c] represents a signed 16-bit offset for thecurrent color component c, and G[c] represents a gain value for thecolor component c. In one embodiment, the gain G[c] may be a 16-bitunsigned number with 2 integer bits and 14 fraction bits (e.g., 2.14 infloating point representation), and the gain G[c] may be applied withrounding. By way of example only, the gain G[c] may have a range ofbetween 0 to 4× (e.g., 4 times the input pixel value).

Next, as shown by Equation 10 below, the computed value Y, which issigned, may then be then clipped to a minimum and maximum range:Y=(Y<min[c])?min[c]:(Y>max[c]) ?max[c]:Y  (10)

The variables min[c] and max[c] may represent signed 16-bit “clippingvalues for the minimum and maximum output values, respectively. In oneembodiment, the BLC logic 232 may also be configured to maintain a countof the number of pixels that were clipped above and below maximum andminimum, respectively, per color component.

Subsequently, the output of the BLC logic 232 is forwarded to the lensshading correction (LSC) logic 234. The LSC logic 234 may be configuredto apply an appropriate gain on a per-pixel basis to compensate fordrop-offs in intensity, which are generally roughly proportional to thedistance from the optical center of the lens 88 of the imaging device30. As can be appreciated, such drop-offs may be the result of thegeometric optics of the lens. By way of example, a lens having idealoptical properties may be modeled as the fourth power of the cosine ofthe incident angle, cos⁴(θ), referred to as the cos⁴ law. However,because lens manufacturing is not perfect, various irregularities in thelens may cause the optical properties to deviate from the assumed cos⁴model. For instance, the thinner edged of the lens usually exhibits themost irregularities. Additionally, irregularities in lens shadingpatterns may also be the result of a microlens array within an imagesensor not being perfectly aligned with the color array filter. Further,the infrared (IR) filter in some lenses may cause the drop-off to beilluminant-dependent and, thus, lens shading gains may be adapteddepending upon the light source detected.

Referring to FIG. 18, a three-dimensional profile depicting lightintensity versus pixel position for a typical lens is illustrated. Asshown, the light intensity near the center 272 of the lens graduallydrops off towards the corners or edges 274 of the lens. The lens shadingirregularities depicted in FIG. 18 may be better illustrated by FIG. 19,which shows a colored drawing of an image 276 that exhibits drop-offs inlight intensity towards the corners and edges. Particularly, it shouldbe noted that the light intensity at the approximate center of the imageappears to be brighter than the light intensity at the corners and/oredges of the image.

In accordance with embodiments of the present techniques, lens shadingcorrection gains may be specified as a two-dimensional grid of gains percolor channel (e.g., Gr, R, B, Gb for a Bayer filter). The gain gridpoints may be distributed at fixed horizontal and vertical intervalswithin the raw frame 154 (FIG. 9). As discussed above in FIG. 9, the rawframe 154 may include an active region 156 which defines an area onwhich processing is performed for a particular image processingoperation. With regard to the lens shading correction operation, anactive processing region, which may be referred to as the LSC region, isdefined within the raw frame region 154. As will be discussed below, theLSC region must be completely inside or at the gain grid boundaries,otherwise results may be undefined.

For instance, referring to FIG. 20, an LSC region 280 and a gain grid282 that may be defined within the raw frame 154 are shown. The LSCregion 280 may have a width 284 and a height 286, and may be defined byan x-offset 288 and a y-offset 290 with respect to the boundary of theraw frame 154. Grid offsets (e.g., grid x-offset 292 and grid y-offset294) from the base 296 of the grid gains 282 to the first pixel 298 inthe LSC region 280 is also provided. These offsets may be within thefirst grid interval for a given color component. The horizontal(x-direction) and vertical (y-direction) grid point intervals 300 and302, respectively, may be specified independently for each colorchannel.

As discussed above, assuming the use of a Bayer color filter array, 4color channels of grid gains (R, B, Gr, and Gb) may be defined. In oneembodiment, a total of 4K (4096) grid points may be available, and foreach color channel, a base address for the start location of grid gainsmay be provided, such as by using a pointer. Further, the horizontal(300) and vertical (302) grid point intervals may be defined in terms ofpixels at the resolution of one color plane and, in certain embodiments,may be provide for grid point intervals separated by a power of 2, suchas by 8, 16, 32, 64, or 128, etc., in horizontal and verticaldirections. As can be appreciated, by utilizing a power of 2, efficientimplementation of gain interpolation using a shift (e.g., division) andadd operations may be achieved. Using these parameters, the same gainvalues can be used even as the image sensor cropping region is changing.For instance, only a few parameters need to be updated to align the gridpoints to the cropped region (e.g., updating the grid offsets 300 and302) instead of updating all grid gain values. By way of example only,this may be useful when cropping is used during digital zoomingoperations. Further, while the gain grid 282 shown in the embodiment ofFIG. 20 is depicted as having generally equally spaced grid points, itshould be understood that in other embodiments, the grid points may notnecessarily be equally spaced. For instance, in some embodiments, thegrid points may be distributed unevenly (e.g., logarithmically), suchthat the grid points are less concentrated in the center of the LSCregion 280, but more concentrated towards the corners of the LSC region280, typically where lens shading distortion is more noticeable.

In accordance with the presently disclosed lens shading correctiontechniques, when a current pixel location is located outside of the LSCregion 280, no gain is applied (e.g., the pixel is passed unchanged).When the current pixel location is at a gain grid location, the gainvalue at that particular grid point may be used. However, when a currentpixel location is between grid points, the gain may be interpolatedusing bi-linear interpolation. An example of interpolating the gain forthe pixel location “G” on FIG. 21 is provided below.

As shown in FIG. 21, the pixel G is between the grid points G0, G1, G2,and G3, which may correspond to the top-left, top-right, bottom-left,and bottom-right gains, respectively, relative to the current pixellocation G. The horizontal and vertical size of the grid interval isrepresented by X and Y, respectively. Additionally, ii and jj representthe horizontal and vertical pixel offsets, respectively, relative to theposition of the top left gain G0. Based upon these factors, the gaincorresponding to the position G may thus be interpolated as follows:

$\begin{matrix}{G = \frac{\begin{matrix}{\left( {G\; 0\left( {Y - {jj}} \right)\left( {X - {ii}} \right)} \right) + \left( {G\; 1\left( {Y - {jj}} \right)({ii})} \right) +} \\{\left( {G\; 2({jj})\left( {X - {ii}} \right)} \right) + \left( {G\; 3({ii})({jj})} \right)}\end{matrix}}{XY}} & \left( {11a} \right)\end{matrix}$The terms in Equation 11a above may then be combined to obtain thefollowing expression:

$\begin{matrix}{G = \frac{\begin{matrix}{{G\;{0\left\lbrack {{XY} - {X({jj})} - {Y({ii})} + {({ii})({jj})}} \right\rbrack}} +} \\{{G\;{1\left\lbrack {{Y({ii})} - {({ii})({jj})}} \right\rbrack}} + {G\;{2\left\lbrack {{X({jj})} - {({ii})({jj})}} \right\rbrack}} + {G\;{3\left\lbrack {({ii})({jj})} \right\rbrack}}}\end{matrix}}{XY}} & \left( {11b} \right)\end{matrix}$In one embodiment, the interpolation method may be performedincrementally, instead of using a multiplier at each pixel, thusreducing computational complexity. For instance, the term (ii)(jj) maybe realized using an adder that may be initialized to 0 at location (0,0) of the gain grid 282 and incremented by the current row number eachtime the current column number increases by a pixel. As discussed above,since the values of X and Y may be selected as powers of two, gaininterpolation may be accomplished using a simple shift operations. Thus,the multiplier is needed only at the grid point G0 (instead of at everypixel), and only addition operations are needed to determine theinterpolated gain for the remaining pixels.

In certain embodiments, the interpolation of gains between the gridpoints may use 14-bit precision, and the grid gains may be unsigned10-bit values with 2 integer bits and 8 fractional bits (e.g., 2.8floating point representation). Using this convention, the gain may havea range of between 0 and 4×, and the gain resolution between grid pointsmay be 1/256.

The lens shading correction techniques may be further illustrated by theprocess 310 shown in FIG. 22. As shown, process 310 begins at step 312,at which the position of a current pixel is determined relative to theboundaries of the LSC region 280 of FIG. 20. Next, decision logic 314determines whether the current pixel position is within the LSC region280. If the current pixel position is outside of the LSC region 280, theprocess 310 continues to step 316, and no gain is applied to the currentpixel (e.g., the pixel passes unchanged).

If the current pixel position is within the LSC region 280, the process310 continues to decision logic 318, at which it is further determinedwhether the current pixel position corresponds to a grid point withinthe gain grid 280. If the current pixel position corresponds to a gridpoint, then the gain value at that grid point is selected and applied tothe current pixel, as shown at step 320. If the current pixel positiondoes not correspond to a grid point, then the process 310 continues tostep 322, and a gain is interpolated based upon the bordering gridpoints (e.g., G0, G1, G2, and G3 of FIG. 21). For instance, theinterpolated gain may be computed in accordance with Equations 11a and11b, as discussed above. Thereafter, the process 310 ends at step 324,at which the interpolated gain from step 322 is applied to the currentpixel.

As will be appreciated, the process 310 may be repeated for each pixelof the image data. For instance, as shown in FIG. 23, athree-dimensional profile depicting the gains that may be applied toeach pixel position within a LSC region (e.g. 280) is illustrated. Asshown, the gain applied at the corners 328 of the image may be generallygreater than the gain applied to the center of the image due to thegreater drop-off in light intensity at the corners, as shown in FIGS. 18and 19. Using the presently described lens shading correctiontechniques, the appearance of light intensity drop-offs in the image maybe reduced or substantially eliminated. For instance, FIG. 24 providesan example of how the colored drawing of image 276 from FIG. 19 mayappear after lens shading correction is applied. As shown, compared tothe original image from FIG. 19, the overall light intensity isgenerally more uniform across the image. Particularly, the lightintensity at the approximate center of the image may be substantiallyequal to the light intensity values at the corners and/or edges of theimage. Additionally, as mentioned above, the interpolated gaincalculation (Equations 11a and 11b) may, in some embodiments, bereplaced with an additive “delta” between grid points by takingadvantage of the sequential column and row incrementing structure. Aswill be appreciated, this reduces computational complexity.

In further embodiments, in addition to using grid gains, a global gainper color component that is scaled as a function of the distance fromthe image center is used. The center of the image may be provided as aninput parameter, and may be estimated by analyzing the light intensityamplitude of each image pixel in the uniformly illuminated image. Theradial distance between the identified center pixel and the currentpixel, may then be used to obtain a linearly scaled radial gain, G_(r),as shown below:G _(r) =G _(p) [c]×R,  (12)wherein G_(p)[c] represents a global gain parameter for each colorcomponent c (e.g., R, B, Gr, and Gb components for a Bayer pattern), andwherein R represents the radial distance between the center pixel andthe current pixel.

With reference to FIG. 25, which shows the LSC region 280 discussedabove, the distance R may be calculated or estimated using severaltechniques. As shown, the pixel C corresponding to the image center mayhave the coordinates (x₀, y₀), and the current pixel G may have thecoordinates (x_(G), y_(G)). In one embodiment, the LSC logic 234 maycalculate the distance R using the following equation:R=√{square root over ((x _(G) −x ₀)²+(y _(G) −y ₀)²)}{square root over((x _(G) −x ₀)²+(y _(G) −y ₀)²)}  (13)

In another embodiment, a simpler estimation formula, shown below, may beutilized to obtain an estimated value for R.R=α×max(abs(x _(G) −x ₀),abs(y _(G) −y ₀))+β×min(abs(x _(G) −x ₀),abs(y_(G) −y ₀))  (14)In Equation 14, the estimation coefficients α and β may be scaled to8-bit values. By way of example only, in one embodiment, α may be equalto approximately 123/128 and β may be equal to approximately 51/128 toprovide an estimated value for R. Using these coefficient values, thelargest error may be approximately 4%, with a median error ofapproximately 1.3%. Thus, even though the estimation technique may besomewhat less accurate than utilizing the calculation technique indetermining R (Equation 13), the margin of error is low enough that theestimated values or R are suitable for determining radial gaincomponents for the present lens shading correction techniques.

The radial gain G_(r) may then be multiplied by the interpolated gridgain value G (Equations 11a and 11b) for the current pixel to determinea total gain that may be applied to the current pixel. The output pixelY is obtained by multiplying the input pixel value X with the totalgain, as shown below:Y=(G×G _(r) ×X)  (15)Thus, in accordance with the present technique, lens shading correctionmay be performed using only the interpolated gain, both the interpolatedgain and the radial gain components. Alternatively, lens shadingcorrection may also be accomplished using only the radial gain inconjunction with a radial grid table that compensates for radialapproximation errors. For example, instead of a rectangular gain grid282, as shown in FIG. 20, a radial gain grid having a plurality of gridpoints defining gains in the radial and angular directions may beprovided. Thus, when determining the gain to apply to a pixel that doesnot align with one of the radial grid points within the LSC region 280,interpolation may be applied using the four grid points that enclose thepixel to determine an appropriate interpolated lens shading gain.

Referring to FIG. 26, the use of interpolated and radial gain componentsin lens shading correction is illustrated by the process 340. It shouldbe noted that the process 340 may include steps that are similar to theprocess 310, described above in FIG. 22. Accordingly, such steps havebeen numbered with like reference numerals. Beginning at step 312, thecurrent pixel is received and its location relative to the LSC region280 is determined. Next, decision logic 314 determines whether thecurrent pixel position is within the LSC region 280. If the currentpixel position is outside of the LSC region 280, the process 340continues to step 316, and no gain is applied to the current pixel(e.g., the pixel passes unchanged). If the current pixel position iswithin the LSC region 280, then the process 340 may continuesimultaneously to step 342 and decision logic 318. Referring first tostep 342, data identifying the center of the image is retrieved. Asdiscussed above, determining the center of the image may includeanalyzing light intensity amplitudes for the pixels under uniformillumination. This may occur during calibration, for instance. Thus, itshould be understood that step 342 does not necessarily encompassrepeatedly calculating the center of the image for processing eachpixel, but may refer to retrieving the data (e.g., coordinates) ofpreviously determined image center. Once the center of the image isidentified, the process 340 may continue to step 344, wherein thedistance between the image center and the current pixel location (R) isdetermined. As discussed above, the value of R may be calculated(Equation 13) or estimated (Equation 14). Then, at step 346, a radialgain component G_(r) may be computed using the distance R and globalgain parameter corresponding to the color component of the current pixel(Equation 12). The radial gain component G_(r) may be used to determinethe total gain, as will be discussed in step 350 below.

Referring back to decision logic 318, a determined whether the currentpixel position corresponds to a grid point within the gain grid 280. Ifthe current pixel position corresponds to a grid point, then the gainvalue at that grid point is determined, as shown at step 348. If thecurrent pixel position does not correspond to a grid point, then theprocess 340 continues to step 322, and an interpolated gain is computedbased upon the bordering grid points (e.g., G0, G1, G2, and G3 of FIG.21). For instance, the interpolated gain may be computed in accordancewith Equations 11a and 11b, as discussed above. Next, at step 350, atotal gain is determined based upon the radial gain determined at step346, as well as one of the grid gains (step 348) or the interpolatedgain (322). As can be appreciated, this may depend on which branchdecision logic 318 takes during the process 340. The total gain is thenapplied to the current pixel, as shown at step 352. Again, it should benoted that like the process 310, the process 340 may also be repeatedfor each pixel of the image data.

The use of the radial gain in conjunction with the grid gains may offervarious advantages. For instance, using a radial gain allows for the useof single common gain grid for all color components. This may greatlyreduce the total storage space required for storing separate gain gridsfor each color component. For instance, in a Bayer image sensor, the useof a single gain grid for each of the R, B, Gr, and Gb components mayreduce the gain grid data by approximately 75%. As will be appreciated,this reduction in grid gain data may decrease implementation costs, asgrid gain data tables may account for a significant portion of memory orchip area in image processing hardware. Further, depending upon thehardware implementation, the use of a single set of gain grid values mayoffer further advantages, such as reducing overall chip area (e.g., suchas when the gain grid values are stored in an on-chip memory) andreducing memory bandwidth requirements (e.g., such as when the gain gridvalues are stored in an off-chip external memory).

Having thoroughly described the functionalities of the lens shadingcorrection logic 234 shown in FIG. 15, the output of the LSC logic 234is subsequently forwarded to the inverse black level compensation (IBLC)logic 236. The IBLC logic 236 provides gain, offset and clipindependently for each color component (e.g., R, B, Gr, and Gb), andgenerally performs the inverse function to the BLC logic 232. Forinstance, as shown by the following operation, the value of the inputpixel is first multiplied by a gain and then offset by a signed value.Y=(X×G[c])+O[c],  (16)wherein X represents the input pixel value for a given color component c(e.g., R, B, Gr, or Gb), O[c] represents a signed 16-bit offset for thecurrent color component c, and G[c] represents a gain value for thecolor component c. In one embodiment, the gain G[c] may have a range ofbetween approximately 0 to 4× (4 times the input pixel value X). Itshould be noted that these variables may be the same variables discussedabove in Equation 9. The computed value Y may be clipped to a minimumand maximum range using, for example, Equation 10. In one embodiment,the IBLC logic 236 may be configured to maintain a count of the numberof pixels that were clipped above and below maximum and minimum,respectively, per color component.

Thereafter, the output of the IBLC logic 236 is received by thestatistics collection block 238, which may provide for the collection ofvarious statistical data points about the image sensor(s) 90, such asthose relating to auto-exposure (AE), auto-white balance (AWB),auto-focus (AF), flicker detection, and so forth. A brief overviewdiscussing the significance of AWB, AE, and AF statistics is providedbelow.

With regard to white balancing, the image sensor response at each pixelmay depend on the illumination source, since the light source isreflected from objects in the image scene. Thus, each pixel valuerecorded in the image scene is related to the color temperature of thelight source. When a white object is illuminated under a low colortemperature, it may appear reddish in the captured image. Conversely, awhite object that is illuminated under a high color temperature mayappear bluish in the captured image. The goal of white balancing is,therefore, to adjust RGB values such that the image appears to the humaneye as if it were taken under canonical light. Thus, in the context ofimaging statistics relating to white balance, color information aboutwhite objects are collected to determine the color temperature of thelight source. In general, white balance algorithms may include two mainsteps. First, the color temperature of the light source is estimated.Second, the estimated color temperature is used to adjust color gainvalues and/or determine/adjust coefficients of a color correctionmatrix. Such gains may be a combination of analog and digital imagesensor gains, as well as ISP digital gains.

Next, auto-exposure generally refers to a process of adjusting pixelintegration time and gains to control the luminance of the capturedimage. For instance, auto-exposure may control the amount of light fromthe scene that is captured by the image sensor(s) by setting theintegration time. Further, auto-focus may refer to determining theoptimal focal length of the lens in order to substantially optimize thefocus of the image. Thus, these various types of statistics, amongothers, may be determined and collected via the statistics collectionblock 238. As shown, the output STATS0 of the statistics collectionblock 238 may be sent to the memory 108 and routed to the control logic84 or, alternatively, may be sent directly to the control logic 84.

As discussed above, the control logic 84 may process the collectedstatistical data to determine one or more control parameters forcontrolling the imaging device 30 and/or the image processing circuitry32. For instance, such the control parameters may include parameters foroperating the lens of the image sensor 90 (e.g., focal length adjustmentparameters), image sensor parameters (e.g., analog and/or digital gains,integration time), as well as ISP pipe processing parameters (e.g.,digital gain values, color correction matrix (CCM) coefficients).Additionally, as mentioned above, in certain embodiments, statisticalprocessing may occur at a precision of 8-bits and, thus, raw pixel datahaving a higher bit-depth may be down-sampled to an 8-bit format forstatistics purposes. As discussed above, down-sampling to 8-bits (or anyother lower-bit resolution) may reduce hardware size (e.g., area) andalso reduce processing complexity, as well as allow for the statisticsdata to be more robust to noise (e.g., using spatial averaging of theimage data).

Before proceeding with a detailed discussion of the ISP pipe logic 82downstream from the ISP front-end logic 80, it should understood thatthe arrangement of various functional logic blocks in the statisticsprocessing units 120 and 122 (e.g., logic blocks 230, 232, 234, 236, and238) and the ISP front-end pixel processing unit 130 (e.g., logic blocks180 and 182) are intended to illustrate only one embodiment of thepresent technique. Indeed, in other embodiments, the logic blocksillustrated herein may be arranged in different ordering, or may includeadditional logic blocks that may perform additional image processingfunctions not specifically described herein. Further, it should beunderstood that the image processing operations performed in thestatistics processing units (e.g., 120 and 122), such as lens shadingcorrection, defective pixel detection/correction, and black levelcompensation, are performed within the statistics processing units forthe purposes of collecting statistical data. Thus, processing operationsperformed upon the image data received by the statistical processingunits are not actually reflected in the image signal 109 (FEProcOut)that is output from the ISP front-end pixel processing logic 130 andforwarded to the ISP pipe processing logic 82.

Before continuing, it should also be noted, that given sufficientprocessing time and the similarity between many of the processingrequirements of the various operations described herein, it is possibleto reconfigure the functional blocks shown herein to perform imageprocessing in a sequential manner, rather than a pipe-lined nature. Aswill be understood, this may further reduce the overall hardwareimplementation costs, but may also increase bandwidth to external memory(e.g., to cache/store intermediate results/data).

The ISP Pipeline (“Pipe”) Processing Logic

Having described the ISP front-end logic 80 in detail above, the presentdiscussion will now shift focus to the ISP pipe processing logic 82.Generally, the function of the ISP pipe logic 82 is to receive raw imagedata, which may be provided from the ISP front-end logic 80 or retrievedfrom memory 108, and to perform additional image processing operations,i.e., prior to outputting the image data to the display device 28.

A block diagram showing an embodiment of the ISP pipe logic 82 isdepicted in FIG. 27. As illustrated, the ISP pipe logic 82 may includeraw processing logic 360, RGB processing logic 362, and YCbCr processinglogic 364. The raw processing logic 360 may perform various imageprocessing operations, such as defective pixel detection and correction,lens shading correction, demosaicing, as well as applying gains forauto-white balance and/or setting a black level, as will be discussedfurther below. As shown in the present embodiment, the input signal 370to the raw processing logic 360 may be the raw pixel output 109 (signalFEProcOut) from the ISP front-end logic 80 or the raw pixel data 112from the memory 108, depending on the present configuration of theselection logic 368.

As a result of demosaicing operations performed within the rawprocessing logic 360, the image signal output 372 may be in the RGBdomain, and may be subsequently forwarded to the RGB processing logic362. For instance, as shown in FIG. 27, the RGB processing logic 362receives the signal 378, which may be the output signal 372 or an RGBimage signal 374 from the memory 108, depending on the presentconfiguration of the selection logic 376. The RGB processing logic 362may provide for various RGB color adjustment operations, including colorcorrection (e.g., using a color correction matrix), the application ofcolor gains for auto-white balancing, as well as global tone mapping, aswill be discussed further below. The RGB processing logic 364 may alsoprovide for the color space conversion of RGB image data to the YCbCr(luma/chroma) color space. Thus, the image signal output 380 may be inthe YCbCr domain, and may be subsequently forwarded to the YCbCrprocessing logic 364.

For instance, as shown in FIG. 27, the YCbCr processing logic 364receives the signal 386, which may be the output signal 380 from the RGBprocessing logic 362 or a YCbCr signal 382 from the memory 108,depending on the present configuration of the selection logic 384. Aswill be discussed in further detail below, the YCbCr processing logic364 may provide for image processing operations in the YCbCr colorspace, including scaling, chroma suppression, luma sharpening,brightness, contrast, and color (BCC) adjustments, YCbCr gamma mapping,chroma decimation, and so forth. The image signal output 390 of theYCbCr processing logic 364 may be sent to the memory 108, or may beoutput from the ISP pipe processing logic 82 as the image signal 114(FIG. 7). The image signal 114 may be sent to the display device 28(either directly or via memory 108) for viewing by the user, or may befurther processed using a compression engine (e.g., encoder 118), aCPU/GPU, a graphics engine, or the like.

In accordance with embodiments of the present techniques, the ISP pipelogic 82 may support the processing of raw pixel data in 8-bit, 10-bit,12-bit, or 14-bit formats. For instance, in one embodiment, 8-bit,10-bit, or 12-bit input data may be converted to 14-bit at the input ofthe raw processing logic 360, and raw processing and RGB processingoperations may be performed with 14-bit precision. In the latterembodiment, the 14-bit image data may be down-sampled to 10 bits priorto the conversion of the RGB data to the YCbCr color space, and theYCbCr processing (logic 364) may be performed with 10-bit precision.

In order to provide a comprehensive description of the various functionsprovided by the ISP pipe processing logic 82, each of the raw processinglogic 360, RGB processing logic 362, and YCbCr processing logic 364, aswell as internal logic for performing various image processingoperations that may be implemented in each respective unit of logic 360,362, and 364, will be discussed sequentially below, beginning with theraw processing logic 360. For instance, referring now to FIG. 28, ablock diagram showing a more detailed view of an embodiment of the rawprocessing logic 360 is illustrated, in accordance with an embodiment ofthe present technique. As shown, the raw processing logic 360 includesthe gain, offset, and clamping (GOC) logic 394, defective pixeldetection/correction (DPDC) logic 396, the noise reduction logic 398,lens shading correction logic 400, GOC logic 402, and demosaicing logic404. Further, while the examples discussed below assume the use of aBayer color filter array with the image sensor(s) 90, it should beunderstood that other embodiments of the present technique may utilizedifferent types of color filters as well.

The input signal 370, which may be a raw image signal, is first receivedby the gain, offset, and clamping (GOC) logic 394. The GOC logic 394 mayprovide similar functions and may be implemented in a similar mannerwith respect to the BLC logic 232 of the statistics processing unit 120of the ISP front-end logic 80, as discussed above in FIG. 15. Forinstance, the GOC logic 394 may provide digital gain, offsets andclamping (clipping) independently for each color component R, B, Gr, andGb of a Bayer image sensor. Particularly, the GOC logic 394 may performauto-white balance or set the black level of the raw image data.Further, in some embodiments, the GOC logic 394 may also be used corrector compensate for an offset between the Gr and Gb color components.

In operation, the input value for the current pixel is first offset by asigned value and multiplied by a gain. This operation may be performedusing the formula shown in Equation 9 above, wherein X represents theinput pixel value for a given color component R, B, Gr, or Gb, O[c]represents a signed 16-bit offset for the current color component c, andG[c] represents a gain value for the color component c. The values forG[c] may be previously determined during statistics processing (e.g., inthe ISP front-end block 80). In one embodiment, the gain G[c] may be a16-bit unsigned number with 2 integer bits and 14 fraction bits (e.g.,2.14 floating point representation), and the gain G[c] may be appliedwith rounding. By way of example only, the gain G[c] may have a range ofbetween 0 to 4×.

The computed pixel value Y (which includes the gain G[c] and offset O[c]from Equation 9 is then be clipped to a minimum and a maximum range inaccordance with Equation 10. As discussed above, the variables min[c]and max[c] may represent signed 16-bit “clipping values” for the minimumand maximum output values, respectively. In one embodiment, the GOClogic 394 may also be configured to maintain a count of the number ofpixels that were clipped above and below maximum and minimum ranges,respectively, for each color component.

Subsequently, the output of the GOC logic 394 is forwarded to thedefective pixel detection and correction logic 396. As discussed abovewith reference to FIG. 15 (DPDC logic 230), defective pixels mayattributable to a number of factors, and may include “hot” (or leaky)pixels, “stuck” pixels, and “dead pixels, wherein hot pixels exhibit ahigher than normal charge leakage relative to non-defective pixels, andthus may appear brighter than non-defective pixel, and wherein a stuckpixel appears as always being on (e.g., fully charged) and thus appearsbrighter, whereas a dead pixel appears as always being off. As such, itmay be desirable to have a pixel detection scheme that is robust enoughto identify and address different types of failure scenarios.Particularly, when compared to the front-end DPDC logic 230, which mayprovide only dynamic defect detection/correction, the pipe DPDC logic396 may provide for fixed or static defect detection/correction, dynamicdefect detection/correction, as well as speckle removal.

In accordance with embodiments of the presently disclosed techniques,defective pixel correction/detection performed by the DPDC logic 396 mayoccur independently for each color component (e.g., R, B, Gr, and Gb),and may include various operations for detecting defective pixels, aswell as for correcting the detected defective pixels. For instance, inone embodiment, the defective pixel detection operations may provide forthe detection of static defects, dynamics defects, as well as thedetection of speckle, which may refer to the electrical interferences ornoise (e.g., photon noise) that may be present in the imaging sensor. Byanalogy, speckle may appear on an image as seemingly random noiseartifacts, similar to the manner in which static may appear on adisplay, such as a television display. Further, as noted above, dynamicdefection correction is regarded as being dynamic in the sense that thecharacterization of a pixel as being defective at a given time maydepend on the image data in the neighboring pixels. For example, a stuckpixel that is always on maximum brightness may not be regarded as adefective pixel if the location of the stuck pixel is in an area of thecurrent image that is dominate by bright white colors. Conversely, ifthe stuck pixel is in a region of the current image that is dominated byblack or darker colors, then the stuck pixel may be identified as adefective pixel during processing by the DPDC logic 396 and correctedaccordingly.

With regard to static defect detection, the location of each pixel iscompared to a static defect table, which may store data corresponding tothe location of pixels that are known to be defective. For instance, inone embodiment, the DPDC logic 396 may monitor the detection ofdefective pixels (e.g., using a counter mechanism or register) and, if aparticular pixel is observed as repeatedly failing, the location of thatpixel is stored into the static defect table. Thus, during static defectdetection, if it is determined that the location of the current pixel isin the static defect table, then the current pixel is identified asbeing a defective pixel, and a replacement value is determined andtemporarily stored. In one embodiment, the replacement value may be thevalue of the previous pixel (based on scan order) of the same colorcomponent. The replacement value may be used to correct the staticdefect during dynamic/speckle defect detection and correction, as willbe discussed below. Additionally, if the previous pixel is outside ofthe raw frame 154 (FIG. 9), then its value is not used, and the staticdefect may be corrected during the dynamic defect correction process.Further, due to memory considerations, the static defect table may storea finite number of location entries. For instance, in one embodiment,the static defect table may be implemented as a FIFO queue configured tostore a total of 16 locations for every two lines of image data. Thelocations in defined in the static defect table will, nonetheless, becorrected using a previous pixel replacement value (rather than via thedynamic defect detection process discussed below). As mentioned above,embodiments of the present technique may also provide for updating thestatic defect table intermittently over time.

Embodiments may provide for the static defect table to be implemented inon-chip memory or off-chip memory. As will be appreciated, using anon-chip implementation may increase overall chip area/size, while usingan off-chip implementation may reduce chip area/size, but increasememory bandwidth requirements. Thus, it should be understood that thestatic defect table may be implemented either on-chip or off-chipdepending on specific implementation requirements, i.e., the totalnumber of pixels that are to be stored within the static defect table.

The dynamic defect and speckle detection processes may be time-shiftedwith respect to the static defect detection process discussed above. Forinstance, in one embodiment, the dynamic defect and speckle detectionprocess may begin after the static defect detection process has analyzedtwo scan lines (e.g., rows) of pixels. As can be appreciated, thisallows for the identification of static defects and their respectivereplacement values to be determined before dynamic/speckle detectionoccurs. For example, during the dynamic/speckle detection process, ifthe current pixel was previously marked as being a static defect, ratherthan applying dynamic/speckle detection operations, the static defect issimply corrected using the previously assessed replacement value.

With regard to dynamic defect and speckle detection, these processes mayoccur sequentially or in parallel. The dynamic defect and speckledetection and correction that is performed by the DPDC logic 396 mayrely on adaptive edge detection using pixel-to-pixel directiongradients. In one embodiment, the DPDC logic 396 may select the eightimmediate neighbors of the current pixel having the same color componentthat are within the raw frame 154 (FIG. 9) are used. In other words, thecurrent pixels and its eight immediate neighbors P0, P1, P2, P3, P4, P5,P6, and P7 may form a 3×3 area, as shown below in FIG. 29.

It should be noted, however, that depending on the location of thecurrent pixel P, pixels outside the raw frame 154 are not consideredwhen calculating pixel-to-pixel gradients. For example, with regard tothe “top-left” case 410 shown in FIG. 29, the current pixel P is at thetop-left corner of the raw frame 154 and, thus, the neighboring pixelsP0, P1, P2, P3, and P5 outside of the raw frame 154 are not considered,leaving only the pixels P4, P6, and P7 (N=3). In the “top” case 412, thecurrent pixel P is at the top-most edge of the raw frame 154 and, thus,the neighboring pixels P0, P1, and P2 outside of the raw frame 154 arenot considered, leaving only the pixels P3, P4, P5, P6, and P7 (N=5).Next, in the “top-right” case 414, the current pixel P is at thetop-right corner of the raw frame 154 and, thus, the neighboring pixelsP0, P1, P2, P4, and P7 outside of the raw frame 154 are not considered,leaving only the pixels P3, P5, and P6 (N=3). In the “left” case 416,the current pixel P is at the left-most edge of the raw frame 154 and,thus, the neighboring pixels P0, P3, and P5 outside of the raw frame 154are not considered, leaving only the pixels P1, P2, P4, P6, and P7(N=5).

In the “center” case 418, all pixels P0-P7 lie within the raw frame 154and are thus used in determining the pixel-to-pixel gradients (N=8). Inthe “right” case 420, the current pixel P is at the right-most edge ofthe raw frame 154 and, thus, the neighboring pixels P2, P4, and P7outside of the raw frame 154 are not considered, leaving only the pixelsP0, P1, P3, P5, and P6 (N=5). Additionally, in the “bottom-left” case422, the current pixel P is at the bottom-left corner of the raw frame154 and, thus, the neighboring pixels P0, P3, P5, P6, and P7 outside ofthe raw frame 154 are not considered, leaving only the pixels P1, P2,and P4 (N=3). In the “bottom” case 424, the current pixel P is at thebottom-most edge of the raw frame 154 and, thus, the neighboring pixelsP5, P6, and P7 outside of the raw frame 154 are not considered, leavingonly the pixels P0, P1, P2, P3, and P4 (N=5). Finally, in the“bottom-right” case 426, the current pixel P is at the bottom-rightcorner of the raw frame 154 and, thus, the neighboring pixels P2, P4,P5, P6, and P7 outside of the raw frame 154 are not considered, leavingonly the pixels P0, P1, and P3 (N=3).

Thus, depending upon the position of the current pixel P, the number ofpixels used in determining the pixel-to-pixel gradients may be 3, 5, or8. In the illustrated embodiment, for each neighboring pixel (k=0 to 7)within the picture boundary (e.g., raw frame 154), the pixel-to-pixelgradients may be calculated as follows:G _(k) =abs(P−P _(k)), for 0≦k≦7 (only for k within the raw frame)  (17)Additionally, an average gradient, G_(av), may be calculated as thedifference between the current pixel and the average, P_(av), of itssurrounding pixels, as shown by the equations below:

$\begin{matrix}{{P_{av} = \frac{\left( {\sum\limits_{k}^{N}P_{k}} \right)}{N}},{{{wherein}\mspace{14mu} N} = 3},5,{{or}\mspace{14mu} 8\mspace{14mu}\left( {{depending}\mspace{14mu}{on}\mspace{14mu}{pixel}\mspace{14mu}{position}} \right)}} & (18) \\{G_{av} = {{abs}\left( {P - P_{av}} \right)}} & (19)\end{matrix}$The pixel-to-pixel gradient values (Equation 17) may be used indetermining a dynamic defect case, and the average of the neighboringpixels (Equations 18 and 19) may be used in identifying speckle cases,as discussed further below.

In one embodiment, dynamic defect detection may be performed by the DPDClogic 396 as follows. First, it is assumed that a pixel is defective ifa certain number of the gradients G_(k) are at or below a particularthreshold, denoted by the variable dynTh (dynamic defect threshold).Thus, for each pixel, a count (C) of the number of gradients forneighboring pixels inside the picture boundaries that are at or belowthe threshold dynTh is accumulated. The threshold dynTh may be acombination of a fixed threshold component and a dynamic thresholdcomponent that may depend on the “activity” present the surroundingpixels. For instance, in one embodiment, the dynamic threshold componentfor dynTh may be determined by calculating a high frequency componentvalue P_(hf) based upon summing the absolute difference between theaverage pixel values P_(av) (Equation 18) and each neighboring pixel, asillustrated below:

$\begin{matrix}{{P_{hf} = {{\frac{8}{N}{\sum\limits_{k}^{N}{{{abs}\left( {P_{av} - P_{k}} \right)}\mspace{14mu}{wherein}\mspace{14mu} N}}} = 3}},5,{{or}\mspace{14mu} 8}} & (20)\end{matrix}$In instances where the pixel is located at an image corner (N=3) or atan image edge (N=5), the P_(hf) may be multiplied by the 8/3 or 8/5,respectively. As can be appreciated, this ensures that the highfrequency component P_(hf) is normalized based on eight neighboringpixels (N=8).

Once P_(hf) is determined, the dynamic defect detection threshold dynThmay be computed as shown below:dynTh=dynTh ₁+(dynTh ₂ ×P _(hf)),  (21)wherein dynTh₁ represents the fixed threshold component, and whereindynTh₂ represents the dynamic threshold component, and is a multiplierfor P_(hf) in Equation 21. A different fixed threshold component dynTh₁may be provided for each color component, but for each pixel of the samecolor, dynTh₁ is the same. By way of example only, dynTh₁ may be set sothat it is at least above the variance of noise in the image.

The dynamic threshold component dynTh₂ may be determined based on somecharacteristic of the image. For instance, in one embodiment, dynTh₂ maybe determined using stored empirical data regarding exposure and/orsensor integration time. The empirical data may be determined duringcalibration of the image sensor (e.g., 90), and may associate dynamicthreshold component values that may be selected for dynTh₂ with each ofa number of data points. Thus, based upon the current exposure and/orsensor integration time value, which may be determined during statisticsprocessing in the ISP front-end logic 80, dynTh₂ may be determined byselecting the dynamic threshold component value from the storedempirical data that corresponds to the current exposure and/or sensorintegration time value. Additionally, if the current exposure and/orsensor integration time value does not correspond directly to one of theempirical data points, then dynTh₂ may be determined by interpolatingthe dynamic threshold component values associated with the data pointsbetween which the current exposure and/or sensor integration time valuefalls. Further, like the fixed threshold component dynTh₁, the dynamicthreshold component dynTh₂ may have different values for each colorcomponent. Thus, composite threshold value dynTh may vary for each colorcomponent (e.g., R, B, Gr, Gb).

As mentioned above, for each pixel, a count C of the number of gradientsfor neighboring pixels inside the picture boundaries that are at orbelow the threshold dynTh is determined. For instance, for eachneighboring pixel within the raw frame 154, the accumulated count C ofthe gradients G_(k) that are at or below the threshold dynTh may becomputed as follows:

$\begin{matrix}{{C = {\sum\limits_{k}^{N}\mspace{14mu}\left( {G_{k} \leq {dynTh}} \right)}},} & (22)\end{matrix}$

for 0≦k≦7 (only for k within the raw frame)

Next, if the accumulated count C is determined to be less than or equalto a maximum count, denoted by the variable dynMaxC, then the pixel maybe considered as a dynamic defect. In one embodiment, different valuesfor dynMaxC may be provided for N=3 (corner), N=5 (edge), and N=8conditions. This logic is expressed below:if (C≦dynMaxC), then the current pixel P is defective.  (23)

As mentioned above, the location of defective pixels may be stored intothe static defect table. In some embodiments, the minimum gradient value(min(G_(k))) calculated during dynamic defect detection for the currentpixel may be stored and may be used to sort the defective pixels, suchthat a greater minimum gradient value indicates a greater “severity” ofa defect and should be corrected during pixel correction before lesssevere defects are corrected. In one embodiment, a pixel may need to beprocessed over multiple imaging frames before being stored into thestatic defect table, such as by filtering the locations of defectivepixels over time. In the latter embodiment, the location of thedefective pixel may be stored into the static defect table only if thedefect appears in a particular number of consecutive images at the samelocation. Further, in some embodiments, the static defect table may beconfigured to sort the stored defective pixel locations based upon theminimum gradient values. For instance, the highest minimum gradientvalue may indicate a defect of greater “severity.” By ordering thelocations in this manner, the priority of static defect correction maybe set, such that the most severe or important defects are correctedfirst. Additionally, the static defect table may be updated over time toinclude newly detected static defects, and ordering them accordinglybased on their respective minimum gradient values.

Speckle detection, which may occur in parallel with the dynamic defectdetection process described above, may be performed by determining ifthe value G_(av) (Equation 19) is above a speckle detection thresholdspkTh. Like the dynamic defect threshold dynTh, the speckle thresholdspkTh may also include fixed and dynamic components, referred to byspkTh₁ and spkTh₂, respectively. In general, the fixed and dynamiccomponents spkTh₁ and spkTh₂ may be set more “aggressively” compared tothe dynTh₁ and dynTh₂ values, in order to avoid falsely detectingspeckle in areas of the image that may be more heavily textured andothers, such as text, foliage, certain fabric patterns, etc.Accordingly, in one embodiment, the dynamic speckle threshold componentspkTh₂ may be increased for high-texture areas of the image, anddecreased for “flatter” or more uniform areas. The speckle detectionthreshold spkTh may be computed as shown below:spkTh=spkTh ₁+(spkTh ₂ ×P _(hf)),  (24)wherein spkTh₁ represents the fixed threshold component, and whereinspkTh₂ represents the dynamic threshold component. The detection ofspeckle may then be determined in accordance with the followingexpression:if (G _(av) >spkTh), then the current pixel P is speckled.  (25)

Once defective pixels have been identified, the DPDC logic 396 may applypixel correction operations depending on the type of defect detected.For instance, if the defective pixel was identified as a static defect,the pixel is replaced with the stored replacement value, as discussedabove (e.g., the value of the previous pixel of the same colorcomponent). If the pixel was identified as either a dynamic defect or asspeckle, then pixel correction may be performed as follows. First,gradients are computed as the sum of the absolute difference between thecenter pixel and a first and second neighbor pixels (e.g., computationof G_(k) of Equation 17) for four directions, a horizontal (h)direction, a vertical (v) direction, a diagonal-positive direction (dp),and a diagonal-negative direction (dn), as shown below:G _(h) =G ₃ +G ₄  (26)G _(v) =G ₁ +G ₆  (27)G _(dp) =G ₂ +G ₅  (28)G _(dn) =G ₂ +G ₇  (29)

Next, the corrective pixel value P_(c) may be determined via linearinterpolation of the two neighboring pixels associated with thedirectional gradient G_(h), G_(v), G_(dp), and G_(dn) that has thesmallest value. For instance, in one embodiment, the logic statementbelow may express the calculation of P_(C):

$\begin{matrix}{{{if}\left( {\min==G_{h}} \right)}{{P_{C} = \frac{P_{3} + P_{4}}{2}};}{{else}\mspace{14mu}{{if}\left( {\min==G_{v}} \right)}}{{P_{C} = \frac{P_{1} + P_{6}}{2}};}{{else}\mspace{14mu}{{if}\left( {\min==G_{dp}} \right)}}{{P_{C} = \frac{P_{2} + P_{5}}{2}};}{{else}\mspace{14mu}{{if}\left( {\min==G_{dn}} \right)}}{{P_{C} = \frac{P_{0} + P_{7}}{2}};}} & (30)\end{matrix}$The pixel correction techniques implemented by the DPDC logic 396 mayalso provide for exceptions at boundary conditions. For instance, if oneof the two neighboring pixels associated with the selected interpolationdirection is outside of the raw frame, then the value of the neighborpixel that is within the raw frame is substituted instead. Thus, usingthis technique, the corrective pixel value will be equivalent to thevalue of the neighbor pixel within the raw frame.

It should be noted that the defective pixel detection/correctiontechniques applied by the DPDC logic 396 during the ISP pipe processingis more robust compared to the DPDC logic 230 in the ISP front-end logic80. As discussed in the embodiment above, the DPDC logic 230 performsonly dynamic defect detection and correction using neighboring pixels inonly the horizontal direction, whereas the DPDC logic 396 provides forthe detection and correction of static defects, dynamic defects, as wellas speckle, using neighboring pixels in both horizontal and verticaldirections.

As will be appreciated, the storage of the location of the defectivepixels using a static defect table may provide for temporal filtering ofdefective pixels with lower memory requirements. For instance, comparedto many conventional techniques which store entire images and applytemporal filtering to identify static defects over time, embodiments ofthe present technique only store the locations of defective pixels,which may typically be done using only a fraction of the memory requiredto store an entire image frame. Further, as discussed above, the storingof a minimum gradient value (min(G_(k))), allows for an efficient use ofthe static defect table prioritizing the order of the locations at whichdefective pixels are corrected (e.g., beginning with those that will bemost visible).

Additionally, the use of thresholds that include a dynamic component(e.g., dynTh₂ and spkTh₂) may help to reduce false defect detections, aproblem often encountered in conventional image processing systems whenprocessing high texture areas of an image (e.g., text, foliage, certainfabric patterns, etc.). Further, the use of directional gradients (e.g.,h, v, dp, dn) for pixel correction may reduce the appearance of visualartifacts if a false defect detection occurs. For instance, filtering inthe minimum gradient direction may result in a correction that stillyields acceptable results under most cases, even in cases of falsedetection. Additionally, the inclusion of the current pixel P in thegradient calculation may improve the accuracy of the gradient detection,particularly in the case of hot pixels.

The above-discussed defective pixel detection and correction techniquesimplemented by the DPDC logic 396 may be summarized by a series of flowcharts provided in FIGS. 30-32. For instance, referring first to FIG.30, a process 430 for detecting static defects is illustrated. Beginninginitially at step 432, an input pixel P is received at a first time, T₀.Next, at step 434, the location of the pixel P is compared to the valuesstored in a static defect table. Decision logic 436 determines whetherthe location of the pixel P is found in the static defect table. If thelocation of P is in the static defect table, then the process 430continues to step 438, wherein the pixel P is marked as a static defectand a replacement value is determined. As discussed above, thereplacement value may be determined based upon the value of the previouspixel (in scan order) of the same color component. The process 430 thencontinues to step 440, at which the process 430 proceeds to the dynamicand speckle detection process 444, illustrated in FIG. 31. Additionally,if at decision logic 436, the location of the pixel P is determined notto be in the static defect table, then the process 430 proceeds to step440 without performing step 438.

Continuing to FIG. 31, the input pixel P is received at time T1, asshown by step 446, for processing to determine whether a dynamic defector speckle is present. Time T1 may represent a time-shift with respectto the static defect detection process 430 of FIG. 30. As discussedabove, the dynamic defect and speckle detection process may begin afterthe static defect detection process has analyzed two scan lines (e.g.,rows) of pixels, thus allowing time for the identification of staticdefects and their respective replacement values to be determined beforedynamic/speckle detection occurs.

The decision logic 448 determines if the input pixel P was previouslymarked as a static defect (e.g., by step 438 of process 430). If P ismarked as a static defect, then the process 444 may continue to thepixel correction process shown in FIG. 32 and may bypass the rest of thesteps shown in FIG. 31. If the decision logic 448 determines that theinput pixel P is not a static defect, then the process continues to step450, and neighboring pixels are identified that may be used in thedynamic defect and speckle process. For instance, in accordance with theembodiment discussed above and illustrated in FIG. 29, the neighboringpixels may include the immediate 8 neighbors of the pixel P (e.g.,P0-P7), thus forming a 3×3 pixel area. Next, at step 452, pixel-to-pixelgradients are calculated with respect to each neighboring pixel withinthe raw frame 154, as described in Equation 17 above. Additionally, anaverage gradient (G_(av)) may be calculated as the difference betweenthe current pixel and the average of its surrounding pixels, as shown inEquations 18 and 19.

The process 444 then branches to step 454 for dynamic defect detectionand to decision logic 464 for speckle detection. As noted above, dynamicdefect detection and speckle detection may, in some embodiments, occurin parallel. At step 454, a count C of the number of gradients that areless than or equal to the threshold dynTh is determined. As describedabove, the threshold dynTh may include fixed and dynamic components and,in one embodiment, may be determined in accordance with Equation 21above. If C is less than or equal to a maximum count, dynMaxC, then theprocess 444 continues to step 460, and the current pixel is marked asbeing a dynamic defect. Thereafter, the process 444 may continue to thepixel correction process shown in FIG. 32, which will be discussedbelow.

Returning back the branch after step 452, for speckle detection, thedecision logic 464 determines whether the average gradient G_(av) isgreater than a speckle detection threshold spkTh, which may also includea fixed and dynamic component. If G_(av) is greater than the thresholdspkTh, then the pixel P is marked as containing speckle at step 466 and,thereafter, the process 444 continues to FIG. 32 for the correction ofthe speckled pixel. Further, if the output of both of the decision logicblocks 456 and 464 are “NO,” then this indicates that the pixel P doesnot contain dynamic defects, speckle, or even static defects (decisionlogic 448). Thus, when the outputs of decision logic 456 and 464 areboth “NO,” the process 444 may conclude at step 458, whereby the pixel Pis passed unchanged, as no defects (e.g., static, dynamic, or speckle)were detected.

Continuing to FIG. 32, a pixel correction process 470 in accordance withthe techniques described above is provided. At step 472, the input pixelP is received from process 444 of FIG. 31. It should be noted that thepixel P may be received by process 470 from step 448 (static defect) orfrom steps 460 (dynamic defect) and 466 (speckle defect). The decisionlogic 474 then determines whether the pixel P is marked as a staticdefect. If the pixel P is a static defect, then the process 470continues and ends at step 476, whereby the static defect is correctedusing the replacement value determined at step 438 (FIG. 30).

If the pixel P is not identified as a static defect, then the process470 continues from decision logic 474 to step 478, and directionalgradients are calculated. For instance, as discussed above withreference to Equations 26-29, the gradients may be computed as the sumof the absolute difference between the center pixel and first and secondneighboring pixels for four directions (h, v, dp, and dn). Next, at step480, the directional gradient having the smallest value is identifiedand, thereafter, decision logic 482 assesses whether one of the twoneighboring pixels associated with the minimum gradient is locatedoutside of the image frame (e.g., raw frame 154). If both neighboringpixels are within the image frame, then the process 470 continues tostep 484, and a pixel correction value (P_(C)) is determined by applyinglinear interpolation to the values of the two neighboring pixels, asillustrated by Equation 30. Thereafter, the input pixel P may becorrected using the interpolated pixel correction value P_(C), as shownat step 492.

Returning to the decision logic 482, if it is determined that one of thetwo neighboring pixels are located outside of the image frame (e.g., rawframe 165), then instead of using the value of the outside pixel (Pout),the DPDC logic 396 may substitute the value of Pout with the value ofthe other neighboring pixel that is inside the image frame (Pin), asshown at step 488. Thereafter, at step 490, the pixel correction valueP_(c) is determined by interpolating the values of Pin and thesubstituted value of Pout. In other words, in this case, P_(C) may beequivalent to the value of Pin. Concluding at step 492, the pixel P iscorrected using the value P_(C). Before continuing, it should beunderstood that the particular defective pixel detection and correctionprocesses discussed herein with reference to the DPDC logic 396 areintended to reflect only one possible embodiment of the presenttechnique. Indeed, depending on design and/or cost constraints, a numberof variations are possible, and features may be added or removed suchthat the overall complexity and robustness of the defectdetection/correction logic is between the simpler detection/correctionlogic 230 implemented in the ISP front-end block 80 and the defectdetection/correction logic discussed here with reference to the DPDClogic 396.

Referring back to FIG. 28, the corrected pixel data is output from theDPDC logic 396 and then received by the noise reduction logic 398 forfurther processing. In one embodiment, the noise reduction logic 398 maybe configured to implements two-dimensional edge-adaptive low passfiltering to reduce noise in the image data while maintaining detailsand textures. The edge-adaptive thresholds may be set (e.g., by thecontrol logic 84) based upon the present lighting levels, such thatfiltering may be strengthened under low light conditions. Further, asbriefly mentioned above with regard to the determination of the dynThand spkTh values, noise variance may be determined ahead of time for agiven sensor so that the noise reduction thresholds can be set justabove noise variance, such that during the noise reduction processing,noise is reduced without significantly affecting textures and details ofthe scene (e.g., avoid/reduce false detections). Assuming a Bayer colorfilter implementation, the noise reduction logic 398 may process eachcolor component Gr, R, B, and Gb independently using a separable 7-taphorizontal filter and a 5-tap vertical filter. In one embodiment, thenoise reduction process may be carried out by correcting fornon-uniformity on the green color components (Gb and Gr), and thenperforming horizontal filtering and vertical filtering.

Green non-uniformity (GNU) is generally characterized by a slightbrightness difference between the Gr and Gb pixels given a uniformlyilluminated flat surface. Without correcting or compensating for thisnon-uniformity, certain artifacts, such as a “maze” artifact, may appearin the full color image after demosaicing. During the greennon-uniformity process may include determining, for each green pixel inthe raw Bayer image data, if the absolute difference between a currentgreen pixel (G1) and the green pixel to the right and below (G2) thecurrent pixel is less than a GNU correction threshold (gnuTh). FIG. 33illustrates the location of the G1 and G2 pixels in a 2×2 area of theBayer pattern. As shown, the color of the pixels bordering G1 may bedepending upon whether the current green pixel is a Gb or Gr pixel. Forinstance, if G1 is Gr, then G2 is Gb, the pixel to the right of G1 is R(red), and the pixel below G1 is B (blue). Alternatively, if G1 is Gb,then G2 is Gr, and the pixel to the right of G1 is B, whereas the pixelbelow G1 is R. If the absolute difference between G1 and G2 is less thanthe GNU correction threshold value, then current green pixel G1 isreplaced by the average of G1 and G2, as shown by the logic below:

$\begin{matrix}{{{if}\left( {{{abs}\left( {{G\; 1} - {G\; 2}} \right)} \leq {gnuTh}} \right)};\mspace{14mu}{{G\; 1} = \frac{{G\; 1} + {G\; 2}}{2}}} & (31)\end{matrix}$As can be appreciated, the application of green non-uniformitycorrection in this manner may help to prevent the G1 and G2 pixels frombeing averaged across edges, thus improving and/or preserving sharpness.

Horizontal filtering is applied subsequent to green non-uniformitycorrection and may, in one embodiment, provide a 7-tap horizontalfilter. Gradients across the edge of each filter tap are computed, andif it is above a horizontal edge threshold (horzTh), the filter tap isfolded to the center pixel, as will be illustrated below. The horizontalfilter may process the image data independently for each color component(R, B, Gr, Gb) and may use unfiltered values as inputs values.

By way of example, FIG. 34 shows a graphical depiction of a set ofhorizontal pixels P0 to P6, with a center tap positioned at P3. Basedupon the pixels shown in FIG. 34, edge gradients for each filter tap maybe calculated as follows:Eh0=abs(P0−P1)  (32)Eh1=abs(P1−P2)  (33)Eh2=abs(P2−P3)  (34)Eh3=abs(P3−P4)  (35)Eh4=abs(P4−P5)  (36)Eh5=abs(P5−P6)  (37)The edge gradients Eh0-Eh5 may then be utilized by the horizontal filtercomponent to determine a horizontal filtering output, P_(horz), usingthe formula shown in Equation 38 below:

$\begin{matrix}{{P_{horz} = {{C\; 0 \times \left\lbrack {{{\left( {{{Eh}\; 2} > {{horzTh}\lbrack c\rbrack}} \right)?P}\; 3} :: {{\left( {{{Eh}\; 1} > {{horzTh}\lbrack c\rbrack}} \right)?P}\; 2} :: {{\left( {{{Eh}\; 0} > {{horzTh}\lbrack c\rbrack}} \right)?P}\; 1} :: {P\; 0}} \right\rbrack} + {C\; 1 \times \left\lbrack {{{\left( {{{Eh}\; 2} > {{horzTh}\lbrack c\rbrack}} \right)?P}\; 3} :: {{\left( {{{Eh}\; 1} > {{horzTh}\lbrack c\rbrack}} \right)?P}\; 2} :: {P\; 1}} \right\rbrack} + {C\; 2 \times \left\lbrack {{{\left( {{{Eh}\; 2} > {{horzTh}\lbrack c\rbrack}} \right)?P}\; 3} :: {P\; 2}} \right\rbrack} + {C\; 3 \times P\; 3} + {C\; 4 \times \left\lbrack {{{\left( {{{Eh}\; 3} > {{horzTh}\lbrack c\rbrack}} \right)?P}\; 3} :: {P\; 4}} \right\rbrack} + {C\; 5 \times \left\lbrack {{{\left( {{{Eh}\; 3} > {{horzTh}\lbrack c\rbrack}} \right)?P}\; 3} :: {{\left( {{{Eh}\; 4} > {{horzTh}\lbrack c\rbrack}} \right)?P}\; 4} :: {P\; 5}} \right\rbrack} + {C\; 6 \times \left\lbrack {{{\left( {{{Eh}\; 3} > {{horzTh}\lbrack c\rbrack}} \right)?P}\; 3} :: {{\left( {{{Eh}\; 4} > {{horzTh}\lbrack c\rbrack}} \right)?P}\; 4} :: {{\left( {{{Eh}\; 5} > {{horzTh}\lbrack c\rbrack}} \right)?P}\; 5} :: {P\; 6}} \right\rbrack}}},} & (38)\end{matrix}$wherein horzTh[c] is the horizontal edge threshold for each colorcomponent c (e.g., R, B, Gr, and Gb), and wherein C0-C6 are the filtertap coefficients corresponding to pixels P0-P6, respectively. Thehorizontal filter output P_(horz) may be applied at the center pixel P3location. In one embodiment, the filter tap coefficients C0-C6 may be16-bit two's complement values with 3 integer bits and 13 fractionalbits (3.13 in floating point). Further, it should be noted that thefilter tap coefficients C0-C6 need not necessarily be symmetrical withrespect to the center pixel P3.

Vertical filtering is also applied by the noise reduction logic 398subsequent to green non-uniformity correction and horizontal filteringprocesses. In one embodiment, the vertical filter operation may providea 5-tap filter, as shown in FIG. 35, with the center tap of the verticalfilter located at P2. The vertical filtering process may occur in asimilar manner as the horizontal filtering process described above. Forinstance, gradients across the edge of each filter tap are computed, andif it is above a vertical edge threshold (vertTh), the filter tap isfolded to the center pixel P2. The vertical filter may process the imagedata independently for each color component (R, B, Gr, Gb) and may useunfiltered values as inputs values.

Based upon the pixels shown in FIG. 35, vertical edge gradients for eachfilter tap may be calculated as follows:Ev0=abs(P0−P1)  (39)Ev1=abs(P1−P2)  (40)Ev2=abs(P2−P3)  (41)Ev3=abs(P3−P4)  (42)The edge gradients Ev0-Ev5 may then be utilized by the vertical filterto determine a vertical filtering output, P_(vert), using the formulashown in Equation 43 below:

$\begin{matrix}{{P_{vert} = {{C\; 0 \times \left\lbrack {{{\left( {{{Ev}\; 1} > {{vertTh}\lbrack c\rbrack}} \right)?P}\; 2} :: {{\left( {{{Ev}\; 0} > {{vertTh}\lbrack c\rbrack}} \right)?P}\; 1} :: {P\; 0}} \right\rbrack} + {C\; 1 \times \left\lbrack {{{\left( {{{Ev}\; 1} > {{vertTh}\lbrack c\rbrack}} \right)?P}\; 2} :: {P\; 1}} \right\rbrack} + {C\; 2 \times P\; 2} + {C\; 3 \times \left\lbrack {{{\left( {{{Ev}\; 2} > {{vertTh}\lbrack c\rbrack}} \right)?P}\; 2} :: {P\; 3}} \right\rbrack} + {C\; 4 \times \left\lbrack {{{\left( {{{Ev}\; 2} > {{vertTh}\lbrack c\rbrack}} \right)?P}\; 2} :: {{\left( {{{Eh}\; 3} > {{vertTh}\lbrack c\rbrack}} \right)?P}\; 3} :: {P\; 4}} \right\rbrack}}},} & (43)\end{matrix}$wherein vertTh[c] is the vertical edge threshold for each colorcomponent c (e.g., R, B, Gr, and Gb), and wherein C0-C4 are the filtertap coefficients corresponding to the pixels P0-P4 of FIG. 35,respectively. The vertical filter output P_(vert) may be applied at thecenter pixel P2 location. In one embodiment, the filter tap coefficientsC0-C4 may be 16-bit two's complement values with 3 integer bits and 13fractional bits (3.13 in floating point). Further, it should be notedthat the filter tap coefficients C0-C4 need not necessarily besymmetrical with respect to the center pixel P2.

Additionally, with regard to boundary conditions, when neighboringpixels are outside of the raw frame 154 (FIG. 9), the values of theout-of-bound pixels are replicated with the value of same color pixel atthe edge of the raw frame. This convention may be implemented for bothhorizontal and vertical filtering operations. By way of example,referring again to FIG. 34, in the case of horizontal filtering, if thepixel P2 is an edge pixel at the left-most edge of the raw frame, andthe pixels P0 and P1 are outside of the raw frame, then the values ofthe pixels P0 and P1 are substituted with the value of the pixel P2 forhorizontal filtering.

Referring again back to the block diagram of the raw processing logic360 shown in FIG. 28, the output of the noise reduction logic 398 issubsequently sent to the lens shading correction (LSC) logic 400 forprocessing. As discussed above, lens shading correction techniques mayinclude applying an appropriate gain on a per-pixel basis to compensatefor drop-offs in light intensity, which may be the result of thegeometric optics of the lens, imperfections in manufacturing,misalignment of the microlens array and the color array filter, and soforth. Further, the infrared (IR) filter in some lenses may cause thedrop-off to be illuminant-dependent and, thus, lens shading gains may beadapted depending upon the light source detected.

In the depicted embodiment, the LSC logic 400 of the ISP pipe 82 may beimplemented in a similar manner, and thus provide generally the samefunctions, as the LSC logic 234 of the ISP front-end block 80, asdiscussed above with reference to FIGS. 18-26. Accordingly, in order toavoid redundancy, it should be understood that the LSC logic 400 of thepresently illustrated embodiment is configured to operate in generallythe same manner as the LSC logic 230 and, as such, the description ofthe lens shading correction techniques provided above will not berepeated here. However, to generally summarize, it should be understoodthat the LSC logic 400 may process each color component of the raw pixeldata stream independently to determine a gain to apply to the currentpixel. In accordance with the above-discussed embodiments, the lensshading correction gain may be determined based upon a defined set ofgain grid points distributed across the imaging frame, wherein theinterval between each grid point is defined by a number of pixels (e.g.,8 pixels, 16 pixels etc.). If the location of the current pixelcorresponds to a grid point, then the gain value associated with thatgrid point is applied to the current pixel. However, if the location ofthe current pixel is between grid points (e.g., G0, G1, G2, and G3 ofFIG. 21), then the LSC gain value may be calculated by interpolation ofthe grid points between which the current pixel is located (Equations11a and 11b). This process is depicted by the process 310 of FIG. 22.Further, as mentioned above with respect to FIG. 20, in someembodiments, the grid points may be distributed unevenly (e.g.,logarithmically), such that the grid points are less concentrated in thecenter of the LSC region 280, but more concentrated towards the cornersof the LSC region 280, typically where lens shading distortion is morenoticeable.

Additionally, as discussed above with reference to FIGS. 25 and 26, theLSC logic 400 may also apply a radial gain component with the grid gainvalues. The radial gain component may be determined based upon distanceof the current pixel from the center of the image (Equations 12-14). Asmentioned, using a radial gain allows for the use of single common gaingrid for all color components, which may greatly reduce the totalstorage space required for storing separate gain grids for each colorcomponent. This reduction in grid gain data may decrease implementationcosts, as grid gain data tables may account for a significant portion ofmemory or chip area in image processing hardware.

Next, referring again to the raw processing logic block diagram 360 ofFIG. 28, the output of the LSC logic 400 is then passed to a secondgain, offset, and clamping (GOC) block 402. The GOC logic 402 may beapplied prior to demosaicing (by logic block 404) and may be used toperform auto-white balance on the output of the LSC logic 400. In thedepicted embodiment, the GOC logic 402 may be implemented in the samemanner as the GOC logic 394 (and the BLC logic 232). Thus, in accordancewith the Equation 9 above, the input received by the GOC logic 402 isfirst offset by a signed value and then multiplied by a gain. Theresulting value is then clipped to a minimum and a maximum range inaccordance with Equation 10.

Thereafter, the output of the GOC logic 402 is forwarded to thedemosaicing logic 404 for processing to produce a full color (RGB) imagebased upon the raw Bayer input data. As will be appreciated, the rawoutput of an image sensor using a color filter array, such as a Bayerfilter is “incomplete” in the sense that each pixel is filtered toacquire only a single color component. Thus, the data collected for anindividual pixel alone is insufficient to determine color. Accordingly,demosaicing techniques may be used to generate a full color image fromthe raw Bayer data by interpolating the missing color data for eachpixel.

Referring now to FIG. 36, a graphical process flow 500 that provides ageneral overview as to how demosaicing may be applied to a raw Bayerimage pattern 502 to produce a full color RGB is illustrated. As shown,a 4×4 portion 504 of the raw Bayer image 502 may include separatechannels for each color component, including a green channel 506, a redchannel 508, and a blue channel 510. Because each imaging pixel in aBayer sensor only acquires data for one color, the color data for eachcolor channel 506, 508, and 510 may be incomplete, as indicated by the“?” symbols. By applying a demosaicing technique 512, the missing colorsamples from each channel may be interpolated. For instance, as shown byreference number 514, interpolated data G′ may be used to fill themissing samples on the green color channel Similarly, interpolated dataR′ may (in combination with the interpolated data G′ 514) be used tofill the missing samples on the red color channel 516, and interpolateddata B′ may (in combination with the interpolated data G′ 514) be usedto fill the missing samples on the blue color channel 518. Thus, as aresult of the demosaicing process, each color channel (R, G, B) willhave a full set of color data, which may then be used to reconstruct afull color RGB image 520.

A demosaicing technique that may be implemented by the demosaicing logic404 will now be described in accordance with one embodiment. On thegreen color channel, missing color samples may be interpolated using alow pass directional filter on known green samples and a high pass (orgradient) filter on the adjacent color channels (e.g., red and blue).For the red and blue color channels, the missing color samples may beinterpolated in a similar manner, but by using low pass filtering onknown red or blue values and high pass filtering on co-locatedinterpolated green values. Further, in one embodiment, demosaicing onthe green color channel may utilize a 5×5 pixel block edge-adaptivefilter based on the original Bayer color data. As will be discussedfurther below, the use of an edge-adaptive filter may provide for thecontinuous weighting based on gradients of horizontal and verticalfiltered values, which reduce the appearance of certain artifacts, suchas aliasing, “checkerboard,” or “rainbow” artifacts, commonly seen inconventional demosaicing techniques.

During demosaicing on the green channel, the original values for thegreen pixels (Gr and Gb pixels) of the Bayer image pattern are used.However, in order to obtain a full set of data for the green channel,green pixel values may be interpolated at the red and blue pixels of theBayer image pattern. In accordance with the present technique,horizontal and vertical energy components, respectively referred to asEh and Ev, are first calculated at red and blue pixels based on theabove-mentioned 5×5 pixel block. The values of Eh and Ev may be used toobtain an edge-weighted filtered value from the horizontal and verticalfiltering steps, as discussed further below.

By way of example, FIG. 37 illustrates the computation of the Eh and Evvalues for a red pixel centered in the 5×5 pixel block at location (j,i), wherein j corresponds to a row and i corresponds to a column. Asshown, the calculation of Eh considers the middle three rows (j−1, j,j+1) of the 5×5 pixel block, and the calculation of Ev considers themiddle three columns (i−1, i, i+1) of the 5×5 pixel block. To computeEh, the absolute value of the sum of each of the pixels in the redcolumns (i−2, i, i+2) multiplied by a corresponding coefficient (e.g.,−1 for columns i−2 and i+2; 2 for column i) is summed with the absolutevalue of the sum of each of the pixels in the blue columns (i−1, i+1)multiplied by a corresponding coefficient (e.g., 1 for column i−1; −1for column i+1). To compute Ev, the absolute value of the sum of each ofthe pixels in the red rows (j−2, j, j+2) multiplied by a correspondingcoefficient (e.g., −1 for rows j−2 and j+2; 2 for row j) is summed withthe absolute value of the sum of each of the pixels in the blue rows(j−1, j+1) multiplied by a corresponding coefficient (e.g., 1 for rowj−1; −1 for row j+1). These computations are illustrated by Equations 44and 45 below:

$\begin{matrix}{{Eh} = {{abs}\left\lbrack {2\left( {\left( {{P\left( {{j - 1},i} \right)} + {P\left( {j,i} \right)} + {P\left( {{j + 1},i} \right)}} \right) - \left( {{P\left( {{j - 1},{i - 2}} \right)} + {P\left( {j,{i - 2}} \right)} + {P\left( {{j + 1},{i - 2}} \right)}} \right) - \left( {{P\left( {{j - 1},{i + 2}} \right)} + {P\left( {j,{i + 2}} \right)} + {P\left( {{j + 1},{i + 2}} \right)}} \right\rbrack + {{abs}\left\lbrack {\left( {{P\left( {{j - 1},{i - 1}} \right)} + {P\left( {j,{i - 1}} \right)} + {P\left( {{j + 1},{i - 1}} \right)}} \right) - \left( {{P\left( {{j - 1},{i + 1}} \right)} + {P\left( {j,{i + 1}} \right)} + {P\left( {{j + 1},{i + 1}} \right)}} \right\rbrack} \right.}} \right.} \right.}} & (44) \\{{Ev} = {{abs}\left\lbrack {{2\left( {{P\left( {j,{i - 1}} \right)} + {P\left( {j,i} \right)} + {P\left( {j,{i + 1}} \right)}} \right)} - \left( {{P\left( {{j - 2},{i - 1}} \right)} + {P\left( {{j - 2},i} \right)} + {P\left( {{j - 2},{i + 1}} \right)}} \right) - \left( {{P\left( {{j + 2},{i - 1}} \right)} + {P\left( {{j + 2},i} \right)} + {P\left( {{j + 2},{i + 1}} \right\rbrack} + {{abs}\left\lbrack {\left( {{P\left( {{j - 1},{i - 1}} \right)} + {P\left( {{j - 1},i} \right)} + {P\left( {{j - 1},{i + 1}} \right)}} \right) - \left( {{P\left( {{j + 1},{i - 1}} \right)} + {P\left( {{j + 1},i} \right)} + {P\left( {{j + 1},{i + 1}} \right)}} \right\rbrack} \right.}} \right.} \right.}} & (45)\end{matrix}$Thus, the total energy sum may be expressed as: Eh+Ev. Further, whilethe example shown in FIG. 37 illustrates the computation of Eh and Evfor a red center pixel at (j, i), it should be understood that the Ehand Ev values may be determined in a similar manner for blue centerpixels.

Next, horizontal and vertical filtering may be applied to the Bayerpattern to obtain the vertical and horizontal filtered values Gh and Gv,which may represent interpolated green values in the horizontal andvertical directions, respectively. The filtered values Gh and Gv may bedetermined using a low pass filter on known neighboring green samples inaddition to using directional gradients of the adjacent color (R or B)to obtain a high frequency signal at the locations of the missing greensamples. For instance, with reference to FIG. 38, an example ofhorizontal interpolation for determining Gh will now be illustrated.

As shown in FIG. 38, five horizontal pixels (R0, G1, R2, G3, and R4) ofa red line 530 of the Bayer image, wherein R2 is assumed to be thecenter pixel at (j, i), may be considered in determining Gh. Filteringcoefficients associated with each of these five pixels are indicated byreference numeral 532. Accordingly, the interpolation of a green value,referred to as G2′, for the center pixel R2, may be determined asfollows:

$\begin{matrix}{{G\; 2^{\prime}} = {\frac{{G\; 1} + {G\; 3}}{2} + \frac{{2R\; 2} - \left( \frac{{R\; 0} + {R\; 2}}{2} \right) - \left( \frac{{R\; 2} + {R\; 4}}{2} \right)}{2}}} & (46)\end{matrix}$Various mathematical operations may then be utilized to produce theexpression for G2′ shown in Equations 47 and 48 below:

$\begin{matrix}{{G\; 2^{\prime}} = {\frac{{2G\; 1} + {2G\; 3}}{4} + \frac{{4R\; 2} - {R\; 0} - {R\; 2} - {R\; 2} - {R\; 4}}{4}}} & (47) \\{{G\; 2^{\prime}} = \frac{{2G\; 1} + {2G\; 3} + {2R\; 2} - {R\; 0} - {R\; 4}}{4}} & (48)\end{matrix}$

Thus, with reference to FIG. 38 and the Equations 46-48 above, thegeneral expression for the horizontal interpolation for the green valueat (j, i) may be derived as:

$\begin{matrix}{{G\; h} = \frac{\begin{pmatrix}{{2{P\left( {j,{i - 1}} \right)}} + {2{P\left( {j,{i + 1}} \right)}} +} \\{{2P\left( {j,i} \right)} - {P\left( {j,{i - 2}} \right)} - {P\left( {j,{i + 2}} \right)}}\end{pmatrix}}{4}} & (49)\end{matrix}$

The vertical filtering component Gv may be determined in a similarmanner as Gh. For example, referring to FIG. 39, five vertical pixels(R0, G1, R2, G3, and R4) of a red column 534 of the Bayer image andtheir respective filtering coefficients 536, wherein R2 is assumed to bethe center pixel at (j, i), may be considered in determining Gv. Usinglow pass filtering on the known green samples and high pass filtering onthe red channel in the vertical direction, the following expression maybe derived for Gv:

$\begin{matrix}{{G\; v} = \frac{\begin{pmatrix}{{2{P\left( {{j - 1},i} \right)}} + {2{P\left( {{j + 1},i} \right)}} +} \\{{2P\left( {j,i} \right)} - {P\left( {{j - 2},i} \right)} - {P\left( {{j + 2},i} \right)}}\end{pmatrix}}{4}} & (50)\end{matrix}$While the examples discussed herein have shown the interpolation ofgreen values on a red pixel, it should be understood that theexpressions set forth in Equations 49 and 50 may also be used in thehorizontal and vertical interpolation of green values for blue pixels.

The final interpolated green value G′ for the center pixel (j, i) may bedetermined by weighting the horizontal and vertical filter outputs (Ghand Gv) by the energy components (Eh and Ev) discussed above to yieldthe following equation:

$\begin{matrix}{{G^{\prime}\left( {j,i} \right)} = {{\left( \frac{Ev}{{Eh} + {Ev}} \right){Gh}} + {\left( \frac{Eh}{{Eh} + {Ev}} \right){Gv}}}} & (51)\end{matrix}$As discussed above, the energy components Eh and Ev may provide foredge-adaptive weighting of the horizontal and vertical filter outputs Ghand Gv, which may help to reduce image artifacts, such as rainbow,aliasing, or checkerboard artifacts, in the reconstructed RGB image.Additionally, the demosaicing logic 404 may provide an option to bypassthe edge-adaptive weighting feature by setting the Eh and Ev values eachto 1, such that Gh and Gv are equally weighted.

In one embodiment, the horizontal and vertical weighting coefficients,shown in Equation 51 above, may be quantized to reduce the precision ofthe weighting coefficients to a set of “coarse” values. For instance, inone embodiment, the weighting coefficients may be quantized to eightpossible weight ratios: 1/8, 2/8, 3/8, 4/8, 5/8, 6/8, 7/8, and 8/8.Other embodiments may quantize the weighting coefficients into 16 values(e.g., 1/16 to 16/16), 32 values (1/32 to 32/32), and so forth. As canbe appreciated, when compared to using full precision values (e.g.,32-bit floating point values), the quantization of the weightcoefficients may reduce the implementation complexity when determiningand applying the weighting coefficients to horizontal and verticalfilter outputs.

In further embodiments, the presently disclosed techniques, in additionto determining and using horizontal and vertical energy components toapply weighting coefficients to the horizontal (Gh) and vertical (Gv)filtered values, may also determine and utilize energy components in thediagonal-positive and diagonal-negative directions. For instance, insuch embodiments, filtering may also be applied in the diagonal-positiveand diagonal-negative directions. Weighting of the filter outputs mayinclude selecting the two highest energy components, and using theselected energy components to weight their respective filter outputs.For example, assuming that the two highest energy components correspondto the vertical and diagonal-positive directions, the vertical anddiagonal-positive energy components are used to weight the vertical anddiagonal-positive filter outputs to determine the interpolated greenvalue (e.g., at a red or blue pixel location in the Bayer pattern).

Next, demosaicing on the red and blue color channels may be performed byinterpolating red and blue values at the green pixels of the Bayer imagepattern, interpolating red values at the blue pixels of the Bayer imagepattern, and interpolating blue values at the red pixels of the Bayerimage pattern. In accordance with the present discussed techniques,missing red and blue pixel values may be interpolated using low passfiltering based upon known neighboring red and blue pixels and high passfiltering based upon co-located green pixel values, which may beoriginal or interpolated values (from the green channel demosaicingprocess discussed above) depending on the location of the current pixel.Thus, with regard to such embodiments, it should be understood thatinterpolation of missing green values may be performed first, such thata complete set of green values (both original and interpolated values)is available when interpolating the missing red and blue samples.

The interpolation of red and blue pixel values may be described withreference to FIG. 40, which illustrates various 3×3 blocks of the Bayerimage pattern to which red and blue demosaicing may be applied, as wellas interpolated green values (designated by G′) that may have beenobtained during demosaicing on the green channel. Referring first toblock 540, the interpolated red value, R′₁₁, for the Gr pixel (G₁₁) maybe determined as follows:

$\begin{matrix}{{R_{11}^{\prime} = {\frac{\left( {R_{10} + R_{12}} \right)}{2} + \frac{\left( {{2G_{11}} - G_{10}^{\prime} - G_{12}^{\prime}} \right)}{2}}},} & (52)\end{matrix}$where G′₁₀ and G′₁₂ represent interpolated green values, as shown byreference number 548. Similarly, the interpolated blue value, B′₁₁, forthe Gr pixel (G₁₁) may be determined as follows:

$\begin{matrix}{{B_{11}^{\prime} = {\frac{\left( {B_{01} + B_{21}} \right)}{2} + \frac{\left( {{2G_{11}} - G_{01}^{\prime} - G_{21}^{\prime}} \right)}{2}}},} & (53)\end{matrix}$wherein G′₀₁ and G′₂₁ represent interpolated green values (548).

Next, referring to the pixel block 542, in which the center pixel is aGb pixel (G₁₁), the interpolated red value, R′₁₁, and blue value B′₁₁,may be determined as shown in Equations 54 and 55 below:

$\begin{matrix}{R_{11}^{\prime} = {\frac{\left( {R_{01} + R_{21}} \right)}{2} + \frac{\left( {{2G_{11}} - G_{01}^{\prime} - G_{21}^{\prime}} \right)}{2}}} & (54) \\{B_{11}^{\prime} = {\frac{\left( {B_{10} + B_{12}} \right)}{2} + \frac{\left( {{2G_{11}} - G_{10}^{\prime} - G_{12}^{\prime}} \right)}{2}}} & (55)\end{matrix}$

Further, referring to pixel block 544, the interpolation of a red valueon a blue pixel, B₁₁, may be determined as follows:

$\begin{matrix}{{R_{11}^{\prime} = {\frac{\left( {R_{00} + R_{02} + R_{20} + R_{22}} \right)}{4} + \frac{\left( {{4G_{11}^{\prime}} - G_{00}^{\prime} - G_{02}^{\prime} - G_{20}^{\prime} - G_{22}^{\prime}} \right)}{4}}},} & (56)\end{matrix}$wherein G′₀₀, G′₀₂, G′₁₁, G′₂₀, and G′₂₂ represent interpolated greenvalues, as shown by reference number 550. Finally, the interpolation ofa blue value on a red pixel, as shown by pixel block 546, may becalculated as follows:

$\begin{matrix}{B_{11}^{\prime} = {\frac{\left( {B_{00} + B_{02} + B_{20} + B_{22}} \right)}{4} + \frac{\left( {{4G_{11}^{\prime}} - G_{00}^{\prime} - G_{02}^{\prime} - G_{20}^{\prime} - G_{22}^{\prime}} \right)}{4}}} & (57)\end{matrix}$

While the embodiment discussed above relied on color differences (e.g.,gradients) for determining red and blue interpolated values, anotherembodiment may provide for interpolated red and blue values using colorratios. For instance, interpolated green values (blocks 548 and 550) maybe used to obtain a color ratio at red and blue pixel locations of theBayer image pattern, and linear interpolation of the ratios may be usedto determine an interpolated color ratio for the missing color sample.The green value, which may be an interpolated or an original value, maybe multiplied by the interpolated color ratio to obtain a finalinterpolated color value. For instance, interpolation of red and bluepixel values using color ratios may be performed in accordance with theformulas below, wherein Equations 58 and 59 show the interpolation ofred and blue values for a Gr pixel, Equations 60 and 61 show theinterpolation of red and blue values for a Gb pixel, Equation 62 showsthe interpolation of a red value on a blue pixel, and Equation 63 showsthe interpolation of a blue value on a red pixel:

$\begin{matrix}{{R_{11}^{\prime} = {G_{11}\frac{\left( \frac{R_{10}}{G_{10}^{\prime}} \right) + \left( \frac{R_{12}}{G_{12}^{\prime}} \right)}{2}}}\left( {R_{11}^{\prime}\mspace{14mu}{interpolated}\mspace{14mu}{when}\mspace{14mu} G_{11}\mspace{14mu}{is}\mspace{14mu} a\mspace{14mu}{Gr}\mspace{14mu}{pixel}} \right)} & (58) \\{{B_{11}^{\prime} = {G_{11}\frac{\left( \frac{B_{01}}{G_{01}^{\prime}} \right) + \left( \frac{B_{21}}{G_{21}^{\prime}} \right)}{2}}}\left( {B_{11}^{\prime}\mspace{14mu}{interpolated}\mspace{14mu}{when}\mspace{14mu} G_{11}\mspace{14mu}{is}\mspace{14mu} a\mspace{14mu}{Gr}\mspace{14mu}{pixel}} \right)} & (59) \\{{R_{11}^{\prime} = {G_{11}\frac{\left( \frac{R_{01}}{G_{01}^{\prime}} \right) + \left( \frac{R_{21}}{G_{21}^{\prime}} \right)}{2}}}\left( {R_{11}^{\prime}\mspace{14mu}{interpolated}\mspace{14mu}{when}\mspace{14mu} G_{11}\mspace{14mu}{is}\mspace{14mu} a\mspace{14mu}{Gb}\mspace{14mu}{pixel}} \right)} & (60) \\{{B_{11}^{\prime} = {G_{11}\frac{\left( \frac{B_{10}}{G_{10}^{\prime}} \right) + \left( \frac{B_{12}}{G_{12}^{\prime}} \right)}{2}}}\left( {B_{11}^{\prime}\mspace{14mu}{interpolated}\mspace{14mu}{when}\mspace{14mu} G_{11}\mspace{14mu}{is}\mspace{14mu} a\mspace{14mu}{Gb}\mspace{14mu}{pixel}} \right)} & (61) \\{{R_{11}^{\prime} = {G_{11}^{\prime}\frac{\left( \frac{R_{00}}{G_{00}^{\prime}} \right) + \left( \frac{R_{02}}{G_{02}^{\prime}} \right) + \left( \frac{R_{20}}{G_{20}^{\prime}} \right) + \left( \frac{R_{22}}{G_{22}^{\prime}} \right)}{4}}}\left( {R_{11}^{\prime}\mspace{14mu}{interpolated}\mspace{14mu}{on}\mspace{14mu} a\mspace{14mu}{blue}\mspace{14mu}{pixel}\mspace{14mu} B_{11}} \right)} & (62) \\{{B_{11}^{\prime} = {G_{11}^{\prime}\frac{\left( \frac{B_{00}}{G_{00}^{\prime}} \right) + \left( \frac{B_{02}}{G_{02}^{\prime}} \right) + \left( \frac{B_{20}}{G_{20}^{\prime}} \right) + \left( \frac{B_{22}}{G_{22}^{\prime}} \right)}{4}}}\left( {B_{11}^{\prime}\mspace{14mu}{interpolated}\mspace{14mu}{on}\mspace{14mu} a\mspace{14mu}{red}\mspace{14mu}{pixel}\mspace{14mu} R_{11}} \right)} & (63)\end{matrix}$

Once the missing color samples have been interpolated for each imagepixel from the Bayer image pattern, a complete sample of color valuesfor each of the red, blue, and green color channels (e.g., 514, 516, and518 of FIG. 36) may be combined to produce a full color RGB image. Forinstance, referring back FIGS. 27 and 28, the output 372 of the rawpixel processing logic 360 may be an RGB image signal in 8, 10, 12 or14-bit formats.

Referring now to FIGS. 41-44, various flow charts illustrating processesfor demosaicing a raw Bayer image pattern in accordance with disclosedembodiments are illustrated. Specifically, the process 560 of FIG. 41depicts the determination of which color components are to beinterpolated for a given input pixel P. Based on the determination byprocess 560, one or more of the process 572 (FIG. 42) for interpolatinga green value, the process 584 (FIG. 43) for interpolating a red value,or the process 598 (FIG. 44) for interpolating a blue value may beperformed (e.g., by the demosaicing logic 404).

Beginning with FIG. 41, the process 560 begins at step 562 when an inputpixel P is received. Decision logic 564 determines the color of theinput pixel. For instance, this may depend on the location of the pixelwithin the Bayer image pattern. Accordingly, if P is identified as beinga green pixel (e.g., Gr or Gb), the process 560 proceeds to step 566 toobtain interpolated red and blue values for P. This may include, forexample, continuing to the processes 584 and 598 of FIGS. 43 and 44,respectively. If P is identified as being a red pixel, then the process560 proceeds to step 568 to obtain interpolated green and blue valuesfor P. This may include further performing the processes 572 and 598 ofFIGS. 42 and 44, respectively. Additionally, if P is identified as beinga blue pixel, then the process 560 proceeds to step 570 to obtaininterpolated green and red values for P. This may include furtherperforming the processes 572 and 584 of FIGS. 42 and 43, respectively.Each of the processes 572, 584, and 598 are described further below.

The process 572 for determining an interpolated green value for theinput pixel P is illustrated in FIG. 42 and includes steps 574-582. Atstep 574, the input pixel P is received (e.g., from process 560). Next,at step 576, a set of neighboring pixels forming a 5×5 pixel block isidentified, with P being the center of the 5×5 block. Thereafter, thepixel block is analyzed to determine horizontal and vertical energycomponents at step 578. For instance, the horizontal and vertical energycomponents may be determined in accordance with Equations 44 and 45 forcalculating Eh and Ev, respectively. As discussed, the energy componentsEh and Ev may be used as weighting coefficients to provide edge-adaptivefiltering and, therefore, reduce the appearance of certain demosaicingartifacts in the final image. At step 580, low pass filtering and highpass filtering as applied in horizontal and vertical directions todetermine horizontal and vertical filtering outputs. For example, thehorizontal and vertical filtering outputs, Gh and Gv, may be calculatedin accordance with Equations 49 and 50. Next the process 560 continuesto step 582, at which the interpolated green value G′ is interpolatedbased on the values of Gh and Gv weighted with the energy components Ehand Ev, as shown in Equation 51.

Next, with regard to the process 584 of FIG. 43, the interpolation ofred values may begin at step 586, at which the input pixel P is received(e.g., from process 560). At step 588, a set of neighboring pixelsforming a 3×3 pixel block is identified, with P being the center of the3×3 block. Thereafter, low pass filtering is applied on neighboring redpixels within the 3×3 block at step 590, and high pass filtering isapplied on co-located green neighboring values, which may be originalgreen values captured by the Bayer image sensor, or interpolated values(e.g., determined via process 572 of FIG. 42). The interpolated redvalue R′ for P may be determined based on the low pass and high passfiltering outputs, as shown at step 594. Depending on the color of P, R′may be determined in accordance with one of the Equations 52, 54, or 56.

With regard to the interpolation of blue values, the process 598 of FIG.44 may be applied. The steps 600 and 602 are generally identical to thesteps 586 and 588 of the process 584 (FIG. 43). At step 604, low passfiltering is applied on neighboring blue pixels within the 3×3, and, atstep 606, high pass filtering is applied on co-located green neighboringvalues, which may be original green values captured by the Bayer imagesensor, or interpolated values (e.g., determined via process 572 of FIG.42). The interpolated blue value B′ for P may be determined based on thelow pass and high pass filtering outputs, as shown at step 608.Depending on the color of P, B′ may be determined in accordance with oneof the Equations 53, 55, or 57. Further, as mentioned above, theinterpolation of red and blue values may be determined using colordifferences (Equations 52-27) or color ratios (Equations 58-63). Again,it should be understood that interpolation of missing green values maybe performed first, such that a complete set of green values (bothoriginal and interpolated values) is available when interpolating themissing red and blue samples. For example, the process 572 of FIG. 42may be applied to interpolate all missing green color samples beforeperforming the processes 584 and 598 of FIGS. 43 and 44, respectively.

Referring to FIGS. 45-48, examples of colored drawings of imagesprocessed by the raw pixel processing logic 360 in the ISP pipe 82 areprovided. FIG. 45 depicts an original image scene 620, which may becaptured by the image sensor 90 of the imaging device 30. FIG. 46 showsa raw Bayer image 622 which may represent the raw pixel data captured bythe image sensor 90. As mentioned above, conventional demosaicingtechniques may not provide for adaptive filtering based on the detectionof edges (e.g., borders between areas of two or more colors) in theimage data, which may, undesirably, produce artifacts in the resultingreconstructed full color RGB image. For instance, FIG. 47 shows an RGBimage 624 reconstructed using conventional demosaicing techniques, andmay include artifacts, such as “checkerboard” artifacts 626 at the edge628. However, comparing the image 624 to the RGB image 630 of FIG. 48,which may be an example of an image reconstructed using the demosaicingtechniques described above, it can be seen that the checkerboardartifacts 626 present in FIG. 47 are not present, or at least theirappearance is substantially reduced at the edge 628. Thus, the imagesshown in FIGS. 45-48 are intended to illustrate at least one advantagethat the demosaicing techniques disclosed herein have over conventionalmethods.

Referring back to FIG. 27, having now thoroughly described the operationof the raw pixel processing logic 360, which may output an RGB imagesignal 372, the present discussion will now focus on describing theprocessing of the RGB image signal 372 by the RGB processing logic 362.As shown the RGB image signal 372 may be sent to the selection logic 376and/or to the memory 108. The RGB processing logic 362 may receive theinput signal 378, which may be RGB image data from the signal 372 orfrom the memory 108, as shown by signal 374, depending on theconfiguration of the selection logic 376. The RGB image data 378 may beprocessed by the RGB processing logic 362 to perform color adjustmentsoperations, including color correction (e.g., using a color correctionmatrix), the application of color gains for auto-white balancing, aswell as global tone mapping, and so forth.

A block diagram depicting a more detailed view of an embodiment of theRGB processing logic 362 is illustrated in FIG. 49. As shown, the RGBprocessing logic 362 includes the gain, offset, and clamping (GOC) logic640, the RGB color correction logic 642, the GOC logic 644, the RGBgamma adjustment logic, and the color space conversion logic 648. Theinput signal 378 is first received by the gain, offset, and clamping(GOC) logic 640. In the illustrated embodiment, the GOC logic 640 mayapply gains to perform auto-white balancing on one or more of the R, G,or B color channels before processing by the color correction logic 642.

The GOC logic 640 may be similar to the GOC logic 394 of the raw pixelprocessing logic 360, except that the color components of the RGB domainare processed, rather the R, B, Gr, and Gb components of the Bayer imagedata. In operation, the input value for the current pixel is firstoffset by a signed value O[c] and multiplied by a gain G[c], as shown inEquation 9 above, wherein c represents the R, G, and B. As discussedabove, the gain G[c] may be a 16-bit unsigned number with 2 integer bitsand 14 fraction bits (e.g., 2.14 floating point representation), and thevalues for the gain G[c] may be previously determined during statisticsprocessing (e.g., in the ISP front-end block 80). The computed pixelvalue Y (based on Equation 9) is then be clipped to a minimum and amaximum range in accordance with Equation 10. As discussed above, thevariables min[c] and max[c] may represent signed 16-bit “clippingvalues” for the minimum and maximum output values, respectively. In oneembodiment, the GOC logic 640 may also be configured to maintain a countof the number of pixels that were clipped above and below maximum andminimum, respectively, for each color component R, G, and B.

The output of the GOC logic 640 is then forwarded to the colorcorrection logic 642. In accordance with the presently disclosedtechniques, the color correction logic 642 may be configured to applycolor correction to the RGB image data using a color correction matrix(CCM). In one embodiment, the CCM may be a 3×3 RGB transform matrix,although matrices of other dimensions may also be utilized in otherembodiments (e.g., 4×3, etc.). Accordingly, the process of performingcolor correction on an input pixel having R, G, and B components may beexpressed as follows:

$\begin{matrix}{{\begin{bmatrix}R^{\prime} & G^{\prime} & B^{\prime}\end{bmatrix} = {\begin{bmatrix}{C\; C\; M\; 00} & {C\; C\; M\; 01} & {C\; C\; M\; 02} \\{C\; C\; M\; 10} & {C\; C\; M\; 11} & {C\; C\; M\; 12} \\{C\; C\; M\; 20} & {C\; C\; M\; 21} & {C\; C\; M\; 22}\end{bmatrix} \times \begin{bmatrix}R & G & B\end{bmatrix}}},} & (64)\end{matrix}$wherein R, G, and B represent the current red, green, and blue valuesfor the input pixel, CCM00-CCM22 represent the coefficients of the colorcorrection matrix, and R′, G′, and B′ represent the corrected red,green, and blue values for the input pixel. Accordingly, the correctcolor values may be computed in accordance with Equations 65-67 below:R′=(CCM00×R)+(CCM01×G)+(CCM02×B)  (65)G′=(CCM10×R)+(CCM11×G)+(CCM12×B)  (66)B′=(CCM20×R)+(CCM21×G)+(CCM22×B)  (67)

The coefficients (CCM00-CCM22) of the CCM may be determined duringstatistics processing in the ISP front-end block 80, as discussed above.In one embodiment, the coefficients for a given color channel may beselected such that the sum of those coefficients (e.g., CCM00, CCM01,and CCM02 for red color correction) is equal to 1, which may help tomaintain the brightness and color balance. Further, the coefficients aretypically selected such that a positive gain is applied to the colorbeing corrected. For instance, with red color correction, thecoefficient CCM00 may be greater than 1, while one or both of thecoefficients CCM01 and CCM02 may be less than 1. Setting thecoefficients in this manner may enhance the red (R) component in theresulting corrected R′ value while subtracting some of the blue (B) andgreen (G) component. As will be appreciated, this may address issueswith color overlap that may occur during acquisition of the originalBayer image, as a portion of filtered light for a particular coloredpixel may “bleed” into a neighboring pixel of a different color. In oneembodiment, the coefficients of the CCM may be provided as 16-bittwo's-complement numbers with 4 integer bits and 12 fraction bits(expressed in floating point as 4.12). Additionally, the colorcorrection logic 642 may provide for clipping of the computed correctedcolor values if the values exceed a maximum value or are below a minimumvalue.

The output of the RGB color correction logic 642 is then passed toanother GOC logic block 644. The GOC logic 644 may be implemented in anidentical manner as the GOC logic 640 and, thus, a detailed descriptionof the gain, offset, and clamping functions provided will not berepeated here. In one embodiment, the application of the GOC logic 644subsequent to color correction may provide for auto-white balance of theimage data based on the corrected color values, and may also adjustsensor variations of the red-to-green and blue-to-green ratios.

Next, the output of the GOC logic 644 is sent to the RGB gammaadjustment logic 646 for further processing. For instance, the RGB gammaadjustment logic 646 may provide for gamma correction, tone mapping,histogram matching, and so forth. In accordance with disclosedembodiments, the gamma adjustment logic 646 may provide for a mapping ofthe input RGB values to corresponding output RGB values. For instance,the gamma adjustment logic may provide for a set of three lookup tables,one table for each of the R, G, and B components. By way of example,each lookup table may be configured to store 256 entries of 10-bitvalues, each value representing an output level. The table entries maybe evenly distributed in the range of the input pixel values, such thatwhen the input value falls between two entries, the output value may belinearly interpolated. In one embodiment, each of the three lookuptables for R, G, and B may be duplicated, such that the lookup tablesare “double buffered” in memory, thus allowing for one table to be usedduring processing, while its duplicate is being updated. Based on the10-bit output values discussed above, it should be noted that the 14-bitRGB image signal is effectively down-sampled to 10 bits as a result ofthe gamma correction process in the present embodiment.

The output of the gamma adjustment logic 646 may be sent to the memory108 and/or to the color space conversion logic 648. The color spaceconversion (CSC) logic 648 may be configured to convert the RGB outputfrom the gamma adjustment logic 646 to the YCbCr format, in which Yrepresents a luma component, Cb represents a blue-difference chromacomponent, and Cr represents a red-difference chroma component, each ofwhich may be in a 10-bit format as a result of bit-depth conversion ofthe RGB data from 14-bits to 10-bits during the gamma adjustmentoperation. As discussed above, in one embodiment, the RGB output of thegamma adjustment logic 646 may be down-sampled to 10-bits and thusconverted to 10-bit YCbCr values by the CSC logic 648, which may then beforwarded to the YCbCr processing logic 364, which will be discussedfurther below.

The conversion from the RGB domain to the YCbCr color space may beperformed using a color space conversion matrix (CSCM). For instance, inone embodiment, the CSCM may be a 3×3 transform matrix. The coefficientsof the CSCM may be set in accordance with a known conversion equation,such as the BT.601 and BT.709 standards. Additionally, the CSCMcoefficients may be flexible based on the desired range of input andoutputs. Thus, in some embodiments, the CSCM coefficients may bedetermined and programmed based on data collected during statisticsprocessing in the ISP front-end block 80.

The process of performing YCbCr color space conversion on an RGB inputpixel may be expressed as follows:

$\begin{matrix}{{\begin{bmatrix}Y & {Cb} & {Cr}\end{bmatrix} = {\begin{bmatrix}{C\; S\; C\; M\; 00} & {C\; S\; C\; M\; 01} & {C\; S\; C\; M\; 02} \\{C\; S\; C\; M\; 10} & {C\; S\; C\; M\; 11} & {C\; S\; C\; M\; 12} \\{C\; S\; C\; M\; 20} & {C\; S\; C\; M\; 21} & {C\; S\; C\; M\; 22}\end{bmatrix} \times \begin{bmatrix}R & G & B\end{bmatrix}}},} & (68)\end{matrix}$wherein R, G, and B represent the current red, green, and blue valuesfor the input pixel in 10-bit form (e.g., as processed by the gammaadjustment logic 646), CSCM00-CSCM22 represent the coefficients of thecolor space conversion matrix, and Y, Cb, and Cr represent the resultingluma, and chroma components for the input pixel. Accordingly, the valuesfor Y, Cb, and Cr may be computed in accordance with Equations 69-71below:Y=(CSCM00×R)+(CSCM01×G)+(CSCM02×B)  (69)Cb=(CSCM10×R)+(CSCM11×G)+(CSCM12×B)  (70)Cr=(CSCM20×R)+(CSCM21×G)+(CSCM22×B)  (71)Following the color space conversion operation, the resulting YCbCrvalues may be output from the CSC logic 648 as the signal 380, which maybe processed by the YCbCr processing logic 364, as will be discussedbelow.

In one embodiment, the coefficients of the CSCM may be 16-bittwo's-complement numbers with 4 integer bits and 12 fraction bits(4.12). In another embodiment, the CSC logic 648 may further beconfigured to apply an offset to each of the Y, Cb, and Cr values, andto clip the resulting values to a minimum and maximum value. By way ofexample only, assuming that the YCbCr values are in 10-bit form, theoffset may be in a range of −512 to 512, and the minimum and maximumvalues may be 0 and 1023, respectively.

Referring again back to the block diagram of the ISP pipe logic 82 inFIG. 27, the YCbCr signal 380 may be sent to the selection logic 384and/or to the memory 108. The YCbCr processing logic 364 may receive theinput signal 386, which may be YCbCr image data from the signal 380 orfrom the memory 108, as shown by signal 382, depending on theconfiguration of the selection logic 384. The YCbCr image data 386 maythen be processed by the YCbCr processing logic 364 for luma sharpening,chroma suppression, chroma noise reduction, chroma noise reduction, aswell as brightness, contrast, and color adjustments, and so forth.Further, the YCbCr processing logic 364 may provide for gamma mappingand scaling of the processed image data in both horizontal and verticaldirections.

A block diagram depicting a more detailed view of an embodiment of theYCbCr processing logic 364 is illustrated in FIG. 50. As shown, theYCbCr processing logic 364 includes the image sharpening logic 660, thelogic 662 for adjusting brightness, contrast, and/or color, the YCbCrgamma adjustment logic 664, the chroma decimation logic 668, and thescaling logic 670. The YCbCr processing logic 364 may be configured toprocess pixel data in 4:4:4, 4:2:2, or 4:2:0 formats using 1-plane,2-plane, or 3-plane memory configurations. Further, in one embodiment,the YCbCr input signal 386 may provide luma and chroma information as10-bit values.

As will be appreciated, the reference to 1-plane, 2-plane, or 3-planerefers to the number of imaging planes utilized in picture memory. Forinstance, in a 3-plane format, each of the Y, Cb, and Cr components mayutilize separate respective memory planes. In a 2-plane format, a firstplane may be provided for the luma component (Y), and a second planethat interleaves the Cb and Cr samples may be provided for the chromacomponents (Cb and Cr). In a 1-plane format, a single plane in memory isinterleaved with the luma and chroma samples. Further, with regard tothe 4:4:4, 4:2:2, and 4:2:0 formats, it may be appreciated that the4:4:4 format refers to a sampling format in which each of the threeYCbCr components are sampled at the same rate. In a 4:2:2 format, thechroma components Cb and Cr are sub-sampled at half the sampling rate ofthe luma component Y, thus reducing the resolution of chroma componentsCb and Cr by half in the horizontal direction. Similarly the 4:2:0format subs-samples the chroma components Cb and Cr in both the verticaland horizontal directions.

The processing of the YCbCr information may occur within an activesource region defined within a source buffer, wherein the active sourceregion contains “valid” pixel data. For example, referring to FIG. 51, asource buffer 676 having defined therein an active source region 678 isillustrated. In the illustrated example, the source buffer may representa 4:4:4 1-plane format providing source pixels of 10-bit values. Theactive source region 678 may be specified individually for luma (Y)samples and chroma samples (Cb and Cr). Thus, it should be understoodthat the active source region 678 may actually include multiple activesource regions for the luma and chroma samples. The start of the activesource regions 678 for luma and chroma may be determined based on anoffset from a base address (0,0) 680 of the source buffer. For instance,a starting position (Lm_X, Lm_Y) 682 for the luma active source regionmay be defined by an x-offset 686 and a y-offset 690 with respect to thebase address 680. Similarly, a starting position (Ch_X, Ch_Y) 684 forthe chroma active source region may be defined by an x-offset 688 and ay-offset 692 with respect to the base address 680. It should be notedthat in the present example, the y-offsets 688 and 692 for luma andchroma, respectively, may be equal. Based on the starting position 682,the luma active source region may be defined by a width 694 and a height696, each of which may represent the number of luma samples in the x andy directions, respectively. Additionally, based on the starting position684, the chroma active source region may be defined by a width 698 and aheight 700, each of which may represent the number of chroma samples inthe x and y directions, respectively.

FIG. 52 further provides an example showing how active source regionsfor luma and chroma samples may be determined in a two-plane format. Forinstance, as shown, the luma active source region 678 may be defined ina first source buffer 676 (having the base address 680) by the areaspecified by the width 694 and height 696 with respect to the startingposition 682. A chroma active source region 704 may be defined in asecond source buffer 702 (having the base address 706) as the areaspecified by the width 698 and height 700 relative to the startingposition 684.

With the above points in mind and referring back to FIG. 50, the YCbCrsignal 386 is first received by the image sharpening logic 660. Theimage sharpening logic 660 may be configured to perform picturesharpening and edge enhancement processing to increase texture and edgedetails in the image. As will be appreciated, image sharpening mayimprove the perceived image resolution. However, it is generallydesirable that existing noise in the image is not detected as textureand/or edges, and thus not amplified during the sharpening process.

In accordance with the present technique, the image sharpening logic 660may perform picture sharpening using a multi-scale unsharp mask filteron the luma (Y) component of the YCbCr signal. In one embodiment, two ormore low pass Gaussian filters of difference scale sizes may beprovided. For example, in an embodiment that provides two Gaussianfilters, the output (e.g., Gaussian blurring) of a first Gaussian filterhaving a first radius (x) is subtracted from the output of a secondGaussian filter having a second radius (y), wherein x is greater than y,to generate an unsharp mask. Additional unsharp masks may also beobtained by subtracting the outputs of the Gaussian filters from the Yinput. In certain embodiments, the technique may also provide adaptivecoring threshold comparison operations that may be performed using theunsharp masks such that, based upon the results of the comparison(s),gain amounts may be added to a base image, which may be selected as theoriginal Y input image or the output of one of the Gaussian filters, togenerate a final output.

Referring to FIG. 53, block diagram depicting exemplary logic 710 forperforming image sharpening in accordance with embodiments of thepresently disclosed techniques is illustrated. The logic 710 representsa multi-scale unsharp filtering mask that may be applied to an inputluma image Yin. For instance, as shown, Yin is received and processed bytwo low pass Gaussian filters 712 (G1) and 714 (G2). In the presentexample, the filter 712 may be a 3×3 filter and the filter 714 may be a5×5 filter. It should be appreciated, however, that in additionalembodiments, more than two Gaussian filters, including filters ofdifferent scales may also be used (e.g., 7×7, 9×9, etc.). As will beappreciated, due to the low pass filtering process, the high frequencycomponents, which generally correspond to noise, may be removed from theoutputs of the G1 and G2 to produce “unsharp” images (G1out and G2out).As will be discussed below, using an unsharp input image as a base imageallows for noise reduction as part of the sharpening filter.

The 3×3 Gaussian filter 712 and the 5×5 Gaussian filter 714 may bedefined as shown below:

${G\; 1} = {{\frac{\begin{bmatrix}{G\; 1_{1}} & {G\; 1_{1}} & {G\; 1_{1}} \\{G\; 1_{1}} & {G\; 1_{0}} & {G\; 1_{1}} \\{G\; 1_{1}} & {G\; 1_{1}} & {G\; 1_{1}}\end{bmatrix}}{256}\mspace{14mu} G\; 2} = \frac{\begin{bmatrix}{G\; 2_{2}} & {G\; 2_{2}} & {G\; 2_{2}} & {G\; 2_{2}} & {G\; 2_{2}} \\{G\; 2_{2}} & {G\; 2_{1}} & {G\; 2_{1}} & {G\; 2_{1}} & {G\; 2_{2}} \\{G\; 2_{2}} & {G\; 2_{1}} & {G\; 2_{0}} & {G\; 2_{1}} & {G\; 2_{2}} \\{G\; 2_{2}} & {G\; 2_{1}} & {G\; 2_{1}} & {G\; 2_{1}} & {G\; 2_{2}} \\{G\; 2_{2}} & {G\; 2_{2}} & {G\; 2_{2}} & {G\; 2_{2}} & {G\; 2_{2}}\end{bmatrix}}{256}}$By way of example only, the values of the Gaussian filters G1 and G2 maybe selected in one embodiment as follows:

${G\; 1} = {{\frac{\begin{bmatrix}28 & 28 & 28 \\28 & 32 & 28 \\28 & 28 & 28\end{bmatrix}}{256}\mspace{14mu} G\; 2} = \frac{\begin{bmatrix}9 & 9 & 9 & 9 & 9 \\9 & 12 & 12 & 12 & 9 \\9 & 12 & 16 & 12 & 9 \\9 & 12 & 12 & 12 & 9 \\9 & 9 & 9 & 9 & 9\end{bmatrix}}{256}}$

Based on Yin, G1out, and G2out, three unsharp masks, Sharp1, Sharp2, andSharp3, may be generated. Sharp1 may be determined as the unsharp imageG2out of the Gaussian filter 714 subtracted from the unsharp image G1outof the Gaussian filter 712. Because Sharp1 is essentially the differencebetween two low pass filters, it may be referred to as a “mid band”mask, since the higher frequency noise components are already filteredout in the G1out and G2out unsharp images. Additionally, Sharp2 may becalculated by subtracting G2out from the input luma image Yin, andSharp3 may be calculated by subtracting G1out from the input luma imageYin. As will be discussed below, an adaptive threshold coring scheme maybe applied using the unsharp masks Sharp1, Sharp2, and Sharp3.

Referring to the selection logic 716, a base image may be selected basedupon a control signal UnsharpSel. In the illustrated embodiment, thebase image may be either the input image Yin, or the filtered outputsG1out or G2out. As will be appreciated, when an original images has ahigh noise variance (e.g., almost as high as the signal variance), usingthe original image Yin as the base image in sharpening may notsufficiently provide for reduction of the noise components duringsharpening. Accordingly, when a particular threshold of noise content isdetected in the input image, the selection logic 716 may be adapted toselect one of the low pass filtered outputs G1out or G2out from whichhigh frequency content, which may include noise, has been reduced. Inone embodiment, the value of the control signal UnsharpSel may bedetermined by analyzing statistical data acquired during statisticsprocessing in the ISP front-end block 80 to determine the noise contentof the image. By way of example, if the input image Yin has a low noisecontent, such that the appearance noise will likely not increase as aresult of the sharpening process, the input image Yin may be selected asthe base image (e.g., UnsharpSel=j). If the input image Yin isdetermined to contain a noticeable level of noise, such that thesharpening process may amplify the noise, one of the filtered imagesG1out or G2out may be selected (e.g., UnsharpSel=1 or 2, respectively).Thus, by applying an adaptive technique for selecting a base image, thelogic 710 essentially provides a noise reduction function.

Next, gains may be applied to one or more of the Sharp1, Sharp2, andSharp3 masks in accordance with an adaptive coring threshold scheme, asdescribed below. Next, the unsharp values Sharp1, Sharp2, and Sharp3 maybe compared to various thresholds SharpThd1, SharpThd2, and SharpThd3(not necessarily respectively) by way of the comparator blocks 718, 720,and 722. For instance, Sharp1 value is always compared to SharpThd1 atthe comparator block 718. With respective to the comparator block 720,the threshold SharpThd2 may be compared against either Sharp1 or Sharp2,depending upon the selection logic 726. For instance, the selectionlogic 726 may select Sharp1 or Sharp2 depending on the state of acontrol signal SharpCmp2 (e.g., SharpCmp2=1 selects Sharp1; SharpCmp2=0selects Sharp2). For example, in one embodiment, the state of SharpCmp2may be determined depending on the noise variance/content of the inputimage (Yin).

In the illustrated embodiment, it is generally preferable to set theSharpCmp2 and SharpCmp3 values to select Sharp1, unless it is detectedthat the image data has relatively low amounts of noise. This is becauseSharp1, being the difference between the outputs of the Gaussian lowpass filters G1 and G2, is generally less sensitive to noise, and thusmay help reduce the amount to which SharpAmt1, SharpAmt2, and SharpAmt3values vary due to noise level fluctuations in “noisy” image data. Forinstance, if the original image has a high noise variance, some of thehigh frequency components may not be caught when using fixed thresholdsand, thus, may be amplified during the sharpening process. Accordingly,if the noise content of the input image is high, then some of the noisecontent may be present in Sharp2. In such instances, SharpCmp2 may beset to 1 to select the mid-band mask Sharp1 which, as discussed above,has reduced high frequency content due to being the difference of twolow pass filter outputs and is thus less sensitive to noise.

As will be appreciated, a similar process may be applied to theselection of either Sharp1 or Sharp3 by the selection logic 724 underthe control of SharpCmp3. In one embodiment, SharpCmp2 and SharpCmp3 maybe set to 1 by default (e.g., use Sharp1), and set to 0 only for thoseinput images that are identified as having generally low noisevariances. This essentially provides an adaptive coring threshold schemein which the selection of the comparison value (Sharp1, Sharp2, orSharp3) is adaptive based upon the noise variance of an input image.

Based on the outputs of the comparator blocks 718, 720, and 722, thesharpened output image Ysharp may be determined by applying gainedunsharp masks to the base image (e.g., selected via logic 716). Forinstance, referring first to the comparator block 722, SharpThd3 iscompared to the B-input provided by selection logic 724, which shall bereferred to herein as “SharpAbs,” and may be equal to either Sharp1 orSharp3 depending on the state of SharpCmp3. If SharpAbs is greater thanthe threshold SharpThd3, then a gain SharpAmt3 is applied to Sharp3, andthe resulting value is added to the base image. If SharpAbs is less thanthe threshold SharpThd3, then an attenuated gain Att3 may be applied. Inone embodiment, the attenuated gain Att3 may be determined as follows:

$\begin{matrix}{{{Att}\; 3} = \frac{{SharpAmt}\; 3 \times {SharpAbs}}{{SharpThd}\; 3}} & (72)\end{matrix}$wherein, SharpAbs is either Sharp1 or Sharp3, as determined by theselection logic 724. The selection of the based image summed with eitherthe full gain (SharpAmt3) or the attenuated gain (Att3) is performed bythe selection logic 728 based upon the output of the comparator block722. As will be appreciated, the use of an attenuated gain may addresssituations in which SharpAbs is not greater than the threshold (e.g.,SharpThd3), but the noise variance of the image is nonetheless close tothe given threshold. This may help to reduce noticeable transitionsbetween a sharp and an unsharp pixel. For instance, if the image data ispassed without the attenuated gain in such circumstance, the resultingpixel may appear as a defective pixel (e.g., a stuck pixel).

Next, a similar process may be applied with respect to the comparatorblock 720. For instance, depending on the state of SharpCmp2, theselection logic 726 may provide either Sharp1 or Sharp2 as the input tothe comparator block 720 that is compared against the thresholdSharpThd2. Depending on the output of the comparator block 720, eitherthe gain SharpAmt2 or an attenuated gain based upon SharpAmt2, Att2, isapplied to Sharp2 and added to the output of the selection logic 728discussed above. As will be appreciated, the attenuated gain Att2 may becomputed in a manner similar to Equation 72 above, except that the gainSharpAmt2 and the threshold SharpThd2 are applied with respect toSharpAbs, which may be selected as Sharp1 or Sharp2.

Thereafter, a gain SharpAmt1 or an attenuated gain Att1 is applied toSharp1, and the resulting value is summed with output of the selectionlogic 730 to produce the sharpened pixel output Ysharp. The selection ofapplying either the gain SharpAmt1 or attenuated gain Att1 may bedetermined based upon the output of the comparator block 718, whichcompares Sharp1 against the threshold SharpThd1. Again, the attenuatedgain Att1 may be determined in a manner similar to equation 72 above,except that the gain SharpAmt1 and threshold SharpThd1 are applied withrespect to Sharp1. The resulting sharpened pixel values scaled usingeach of the three masks is added to the input pixel Yin to generate thesharpened output Ysharp which, in one embodiment, may be clipped to 10bits (assuming YCbCr processing occurs at 10-bit precision).

As will be appreciated, when compared to conventional unsharp maskingtechniques, the image sharpening techniques set forth in this disclosuremay provide for improving the enhancement of textures and edges whilealso reducing noise in the output image. In particular, the presenttechniques may be well-suited in applications in which images capturedusing, for example, CMOS image sensors, exhibit poor signal-to-noiseratio, such as images acquired under low lighting conditions using lowerresolution cameras integrated into portable devices (e.g., mobilephones). For instance, when the noise variance and signal variance arecomparable, it is difficult to use a fixed threshold for sharpening, assome of the noise components would be sharpened along with texture andedges. Accordingly, the techniques provided herein, as discussed above,may filter the noise from the input image using multi-scale Gaussianfilters to extract features from the unsharp images (e.g., G1out andG2out) in order to provide a sharpened image that also exhibits reducednoise content.

Before continuing, it should be understood that the illustrated logic710 is intended to provide only one exemplary embodiment of the presenttechnique. In other embodiments, additional or fewer features may beprovided by the image sharpening logic 660. For instance, in someembodiments, rather than applying an attenuated gain, the logic 710 maysimply pass the base value. Additionally, some embodiments may notinclude the selection logic blocks 724, 726, or 716. For instance, thecomparator blocks 720 and 722 may simply receive the Sharp2 and Sharp3values, respectively, rather than a selection output from the selectionlogic blocks 724 and 726, respectively. While such embodiments may notprovide for sharpening and/or noise reduction features that are asrobust as the implementation shown in FIG. 53, it should be appreciatedthat such design choices may be the result of cost and/or businessrelated constraints.

In the present embodiment, the image sharpening logic 660 may alsoprovide for edge enhancement and chroma suppression features once thesharpened image output YSharp is obtained. Each of these additionalfeatures will now be discussed below. Referring first to FIG. 54,exemplary logic 738 for performing edge enhancement that may beimplemented downstream from the sharpening logic 710 of FIG. 53 isillustrated in accordance with one embodiment. As shown, the originalinput value Yin is processed by a Sobel filter 740 for edge detection.The Sobel filter 740 may determine a gradient value YEdge based upon a3×3 pixel block (referred to as “A” below) of the original image, withYin being the center pixel of the 3×3 block. In one embodiment, theSobel filter 740 may calculate YEdge by convolving the original imagedata to detect changes in horizontal and vertical directions. Thisprocess is shown below in Equations 73-75.

$\begin{matrix}{S_{x} = {{\begin{bmatrix}1 & 0 & {- 1} \\2 & 0 & {- 2} \\1 & 0 & {- 1}\end{bmatrix}\mspace{14mu} S_{y}} = \begin{bmatrix}1 & 2 & 1 \\0 & 0 & 0 \\{- 1} & {- 2} & {- 1}\end{bmatrix}}} & \; \\{{G_{x} = {S_{x} \times A}},} & (73) \\{{G_{y} = {S_{y} \times A}},} & (74) \\{{{YEdge} = {G_{x} \times G_{y}}},} & (75)\end{matrix}$wherein S_(x) and S_(y) are represent matrix operators for gradientedge-strength detection in the horizontal and vertical directions,respectively, and wherein G_(x) and G_(y) represent gradient images thatcontain horizontal and vertical change derivatives, respectively.Accordingly, the output YEdge is determined as the product of G_(x) andG_(y).

YEdge is then received by selection logic 744 along with the mid-bandSharp1 mask, as discussed above in FIG. 53. Based on the control signalEdgeCmp, either Sharp1 or YEdge is compared to a threshold, EdgeThd, atthe comparator block 742. The state of EdgeCmp may be determined, forexample, based upon the noise content of an image, thus providing anadaptive coring threshold scheme for edge detection and enhancement.Next, the output of the comparator block 742 may be provided to theselection logic 746 and either a full gain or an attenuated gain may beapplied. For instance, when the selected B-input to the comparator block742 (Sharp1 or YEdge) is above EdgeThd, YEdge is multiplied by an edgegain, EdgeAmt, to determine the amount of edge enhancement that is to beapplied. If the B-input at the comparator block 742 is less thanEdgeThd, then an attenuated edge gain, AttEdge, may be applied to avoidnoticeable transitions between the edge enhanced and original pixel. Aswill be appreciated, AttEdge may be calculated in a similar manner asshown in Equation 72 above, but wherein EdgeAmt and EdgeThd are appliedto “SharpAbs,” which may be Sharp1 or YEdge, depending on the output ofthe selection logic 744. Thus, the edge pixel, enhanced using either thegain (EdgeAmt) or the attenuated gain (AttEdge) may be added to YSharp(output of logic 710 of FIG. 53) to obtain the edge-enhanced outputpixel Yout which, in one embodiment, may be clipped to 10 bits (assumingYCbCr processing occurs at 10-bit precision).

With regard to chroma suppression features provided by the imagesharpening logic 660, such features may attenuate chroma at luma edges.Generally, chroma suppression may be performed by applying a chroma gain(attenuation factor) of less than 1 depending on the value (YSharp,Yout) obtained from the luma sharpening and/or edge enhancement stepsdiscussed above. By way of example, FIG. 55 shows a graph 750 thatincludes a curve 752 representing chroma gains that may be selected forcorresponding sharpened luma values (YSharp). The data represented bythe graph 750 may be implemented as a lookup table of YSharp values andcorresponding chroma gains between 0 and 1 (an attenuation factor). Thelookup tables are used to approximate the curve 752. For YSharp valuesthat are co-located between two attenuation factors in the lookup table,linear interpolation may be applied to the two attenuation factorscorresponding to YSharp values above and below the current YSharp value.Further, in other embodiments, the input luma value may also be selectedas one of the Sharp1, Sharp2, or Sharp3 values determined by the logic710, as discussed above in FIG. 53, or the YEdge value determined by thelogic 738, as discussed in FIG. 54.

Next, the output of the image sharpening logic 660 (FIG. 50) isprocessed by the brightness, contrast, and color (BCC) adjustment logic662. A functional block diagram depicting an embodiment of the BCCadjustment logic 662 is illustrated in FIG. 56. As shown the logic 662includes a brightness and contrast processing block 760, global huecontrol block 762, and a saturation control block 764. The presentlyillustrated embodiment provides for processing of the YCbCr data in10-bit precision, although other embodiments may utilize differentbit-depths. The functions of each of blocks 760, 762, and 764 arediscussed below.

Referring first to the brightness and contrast processing block 760, anoffset, YOffset, is first subtracted from the luma (Y) data to set theblack level to zero. This is done to ensure that the contrast adjustmentdoes not alter the black levels. Next, the luma value is multiplied by acontrast gain value to apply contrast control. By way of example, thecontrast gain value may be a 12-bit unsigned with 2 integer bits and 10fractional bits, thus providing for a contrast gain range of up to 4times the pixel value. Thereafter, brightness adjustment may beimplemented by adding (or subtracting) a brightness offset value fromthe luma data. By way of example, the brightness offset in the presentembodiment may be a 10-bit two's complement value having a range ofbetween −512 to +512. Further, it should be noted that brightnessadjustment is performed subsequent to contrast adjustment in order toavoid varying the DC offset when changing contrast. Thereafter, theinitial YOffset is added back to the adjusted luma data to re-positionthe black level.

Blocks 762 and 764 provide for color adjustment based upon huecharacteristics of the Cb and Cr data. As shown, an offset of 512(assuming 10-bit processing) is first subtracted from the Cb and Cr datato position the range to approximately zero. The hue is then adjusted inaccordance with the following equations:Cb _(adj) =Cb cos(θ)+Cr sin(θ),  (76)Cr _(adj) =Cr cos(θ)−Cb sin(θ),  (77)wherein Cb_(adj) and Cr_(adj) represent adjusted Cb and Cr values, andwherein θ represents a hue angle, which may be calculated as follows:

$\begin{matrix}{\theta = {\arctan\left( \frac{Cr}{Cb} \right)}} & (78)\end{matrix}$The above operations are depicted by the logic within the global huecontrol block 762, and may be represented by the following matrixoperation:

$\begin{matrix}{{\begin{bmatrix}{Cb}_{adj} \\{Cr}_{adj}\end{bmatrix} = {\begin{bmatrix}{Ka} & {Kb} \\{- {Kb}} & {Ka}\end{bmatrix}\begin{bmatrix}{Cb} \\{Cr}\end{bmatrix}}},} & (79)\end{matrix}$wherein, Ka=cos(θ), Kb=sin(θ), and θ is defined above in Equation 78.

Next, saturation control may be applied to the Cb_(adj) and Cr_(adj)values, as shown by the saturation control block 764. In the illustratedembodiment, saturation control is performed by applying a globalsaturation multiplier and a hue-based saturation multiplier for each ofthe Cb and Cr values. Hue-based saturation control may improve thereproduction of colors. The hue of the color may be represented in theYCbCr color space, as shown by the color wheel graph 770 in FIG. 57. Aswill be appreciated, the YCbCr hue and saturation color wheel 770 may bederived by shifting the identical color wheel in the HSV color space(hue, saturation, and intensity) by approximately 109 degrees. As shown,the graph 770 includes circumferential values representing thesaturation multiplier (S) within a range of 0 to 1, as well as angularvalues representing θ, as defined above, within a range of between 0 to360°. Each θ may represent a different color (e.g., 49°=magenta,109°=red, 229°=green, etc.). The hue of the color at a particular hueangle θ may be adjusted by selecting an appropriate saturationmultiplier S.

Referring back to FIG. 56, the hue angle θ (calculated in the global huecontrol block 762) may be used as an index for a Cb saturation lookuptable 766 and a Cr saturation lookup table 768. In one embodiment, thesaturation lookup tables 766 and 768 may contain 256 saturation valuesdistributed evenly in the hue range from 0-360° (e.g., the first lookuptable entry is at 0° and the last entry is at 360°) and the saturationvalue S at a given pixel may be determined via linear interpolation ofsaturation values in the lookup table just below and above the currenthue angle θ. A final saturation value for each of the Cb and Crcomponents is obtained by multiplying a global saturation value (whichmay be a global constant for each of Cb and Cr) with the determinedhue-based saturation value. Thus, the final corrected Cb′ and Cr′ valuesmay be determined by multiplying Cb_(adj) and Cr_(adj) with theirrespective final saturation values, as shown in the hue-based saturationcontrol block 764.

Thereafter, the output of the BCC logic 662 is passed to the YCbCr gammaadjustment logic 664, as shown in FIG. 50. In one embodiment, the gammaadjustment logic 664 may provide non-linear mapping functions for the Y,Cb and Cr channels. For instance, the input Y, Cb, and Cr values aremapped to corresponding output values. Again, assuming that the YCbCrdata is processed in 10-bits, an interpolated 10-bit 256 entry lookuptable may be utilized. Three such lookup tables may be provided with onefor each of the Y, Cb, and Cr channels. Each of the 256 input entriesmay be evenly distributed and, an output may be determined by linearinterpolation of the output values mapped to the indices just above andbelow the current input index. In some embodiments, a non-interpolatedlookup table having 1024 entries (for 10-bit data) may also be used, butmay have significantly greater memory requirements. As will beappreciated, by adjusting the output values of the lookup tables, theYCbCr gamma adjustment function may be also be used to perform certainimage filter effects, such as black and white, sepia tone, negativeimages, solarization, and so forth.

Next, chroma decimation may be applied by the chroma decimation logic668 to the output of the gamma adjustment logic 664. In one embodiment,the chroma decimation logic 668 may be configured to perform horizontaldecimation to convert the YCbCr data from a 4:4:4 format to a 4:2:2format, in which the chroma (Cr and Cr) information is sub-sampled athalf rate of the luma data. By way of example only, decimation may beperformed by applying a 7-tap low pass filter, such as a half-bandlanczos filter, to a set of 7 horizontal pixels, as shown below:

$\begin{matrix}{{{Out} = \frac{\begin{matrix}{{C\; 0 \times {{in}\left( {i - 3} \right)}} + {C\; 1 \times {{in}\left( {i - 2} \right)}} + {C\; 2 \times {{in}\left( {i - 1} \right)}} + {C\; 3 \times {{in}(i)}} +} \\{{C\; 4 \times {{in}\left( {i + 1} \right)}} + {C\; 5 \times {{in}\left( {i + 2} \right)}} + {C\; 6 \times {{in}\left( {i + 3} \right)}}}\end{matrix}}{512}},} & (80)\end{matrix}$wherein in(i) represents the input pixel (Cb or Cr), and C0-C6 representthe filtering coefficients of the 7-tap filter. Each input pixel has anindependent filter coefficient (C0-C6) to allow flexible phase offsetfor the chroma filtered samples.

Further, chroma decimation may, in some instances, also be performedwithout filtering. This may be useful when the source image wasoriginally received in 4:2:2 format, but was up-sampled to 4:4:4 formatfor YCbCr processing. In this case, the resulting decimated 4:2:2 imageis identical to the original image.

Subsequently, the YCbCr data output from the chroma decimation logic 668may be scaled using the scaling logic 670 prior to being output from theYCbCr processing block 364. The function of the scaling logic 670 may besimilar to the functionality of the scaling logic 214 in the binningcompensation filter 182 of the front-end pixel processing unit 130, asdiscussed above with reference to FIG. 13. For instance, the scalinglogic 670 may perform horizontal and vertical scaling as two steps. Inone embodiment, a 5-tap polyphase filter may be used for verticalscaling, and a 9-tap polyphase filter may be used for horizontalscaling. The multi-tap polyphase filters may multiply pixels selectedfrom the source image by a weighting factor (e.g., filter coefficient),and then sum the outputs to form the destination pixel. The selectedpixels may be chosen depending on the current pixel position and thenumber of filters taps. For instance, with a vertical 5-tap filter, twoneighboring pixels on each vertical side of a current pixel may beselected and, with a horizontal 9-tap filter, four neighboring pixels oneach horizontal side of the current pixel may be selected. The filteringcoefficients may be provided from a lookup table, and may be determinedby the current between-pixel fractional position. The output 390 of thescaling logic 670 is then output from the YCbCr processing block 364.

Returning back to FIG. 27, the processed output signal 390 may be sentto the memory 108, or may be output from the ISP pipe processing logic82 as the image signal 114 to display hardware (e.g., display 28) forviewing by a user, or to a compression engine (e.g., encoder 118). Insome embodiments, the image signal 114 may be further processed by agraphics processing unit and/or a compression engine and stored beforebeing decompressed and provided to a display. Additionally, one or moreframe buffers may also be provided to control the buffering of the imagedata being output to a display, particularly with respect to video imagedata.

As will be understood, the various image processing techniques describedabove and relating to defective pixel detection and correction, lensshading correction, demosaicing, and image sharpening, among others, areprovided herein by way of example only. Accordingly, it should beunderstood that the present disclosure should not be construed as beinglimited to only the examples provided above. Indeed, the exemplary logicdepicted herein be subject to a number of variations and/or additionalfeatures in other embodiments. Further, it should be appreciated thatthe above-discussed techniques may be implemented in any suitablemanner. For instance, the components of the image processing circuitry32, and particularly the ISP front-end block 80 and the ISP pipe block82 may be implemented using hardware (e.g., suitably configuredcircuitry), software (e.g., via a computer program including executablecode stored on one or more tangible computer readable medium), or viausing a combination of both hardware and software elements.

The specific embodiments described above have been shown by way ofexample, and it should be understood that these embodiments may besusceptible to various modifications and alternative forms. It should befurther understood that the claims are not intended to be limited to theparticular forms disclosed, but rather to cover all modifications,equivalents, and alternatives falling within the spirit and scope ofthis disclosure.

1. A method for processing raw image data using front-end processinglogic of an image signal processor comprising: using an image processor:receiving the raw image data, the raw image data having been acquiredusing an image sensor; processing the raw image data to detect andcorrect defective pixels in the raw image data; processing the raw imagedata to correct for lens shading distortion after detecting andcorrecting the defective pixels, wherein correcting for the lens shadingdistortion comprises applying a lens shading gain to each pixel of theraw image data, the lens shading gain being determined based upon basedupon one or more grid points in a gain grid; after the correcting theraw image data for lens shading distortion, processing the raw imagedata to determine one or more sets of statistics indicative of one ormore properties of the image sensor; applying black level compensationto the raw image data after detecting and correcting the defectivepixels but before correcting the lens shading distortion; and applyinginverse black level compensation to the raw image data after correctingfor the lens shading distortion but before determining the one or moresets of statistics.
 2. The method of claim 1, wherein applying the blacklevel compensation to the raw image data comprises: for each pixel ofthe raw image data: applying an offset to a current pixel; applying adigital gain to the current pixel after applying the offset; and afterapplying the digital gain, applying a clipping factor to the currentpixel if the value of the current pixel is above a particular maximumvalue or below a particular minimum value.
 3. The method of claim 2,wherein the raw image data comprises multiple color components, andwherein the offset, digital gain, and clipping factor are dependent uponthe color component to which the current pixel corresponds.
 4. Themethod of claim 2, wherein applying the inverse black level compensationto the raw image data comprises: for each pixel of the raw image data:applying the digital gain to a current pixel applying an offset to thecurrent pixel after applying the digital gain; and after applying theoffset, applying a clipping factor to the current pixel if the value ofthe current pixel is above a particular maximum value or below aparticular minimum value.
 5. The method of claim 1, wherein the one ormore sets of statistics determined by the front-end processing logiccomprises at least one of auto-exposure statistics, auto-white balancestatistics, auto-focus statistics, flicker detection statistics, or somecombination thereof.
 6. The method of claim 1, wherein the raw imagedata comprises Bayer image data.
 7. The method of claim 1, comprisingconverting the raw image data into a color space corresponding to adisplay device, and displaying the image data on a display device.
 8. Amethod for processing raw image data using front-end processing logic ofan image signal processor comprising: using an image processor:receiving the raw image data, the raw image data having been acquiredusing an image sensor; processing the raw image data to detect andcorrect defective pixels in the raw image data; processing the raw imagedata to correct for lens shading distortion after detecting andcorrecting the defective pixels, wherein correcting for the lens shadingdistortion comprises applying a lens shading gain to each pixel of theraw image data, the lens shading gain being determined based upon basedupon one or more grid points in a gain grid; and after the correctingthe raw image data for lens shading distortion, processing the raw imagedata to determine one or more sets of statistics indicative of one ormore properties of the image sensor; wherein detecting and correctingthe defective pixels in the raw image data comprises: receiving acurrent input pixel of the raw image data; identifying four horizontalneighboring pixels of the same color as the current input pixelincluding two horizontal neighboring pixels located on a first side ofthe current input pixel and two horizontal neighboring pixels located ona second side of the current input pixel; determining pixel-to-pixelgradient values with respect to the current input pixel for each of thefour horizontal neighboring pixels located within a processing regiondefined within an imaging frame of the image sensor; determining a countof the number of gradient values that are less than or equal to adefective pixel threshold value; determining whether the count is lessthan a maximum count value; and if the count is less than the maximumcount value, identifying the current input pixel as a defective pixeland correcting the defective pixel.
 9. The method of claim 8, whereincorrecting the defective pixel comprises: determining the value of afirst neighboring pixel having the same color as the current input pixeland located immediately adjacent to a first side of the input pixel inthe imaging frame, wherein the first neighboring pixel is a pixel thathas already been processed for defective pixel detection and correction;determining whether the first neighboring pixel is within the processingregion of the imaging frame; replacing the value of the current inputpixel with the value of the first neighboring pixel if the firstneighboring pixel is within the processing region; if the firstneighboring pixel is not within the processing region, determining thevalue of a second neighboring pixel having the same color as the currentinput pixel and located immediately adjacent to a second side of theinput pixel in the imaging frame, wherein the second side is oppositethe first side, and wherein the second neighboring pixel is a pixel thathas not yet been processed for defective pixel detection and correction;and replacing the value of the current input pixel with the value of thesecond neighboring pixel.
 10. An imaging processing system comprising: amemory device; and front-end processing logic comprising: a front-endstatistics processing unit configured to receive raw image data acquiredusing an image sensor, to process the raw image data to acquire a set ofstatistics for each frame of the raw image data processed by thefront-end statistics processing unit; and a front-end pixel processingunit configured to process the raw image data for at least one ofreducing noise in the raw image data or scaling the raw image data andto provide the raw image data processed by the front-end pixelprocessing unit to image pipeline processing logic coupled downstreamfrom the front-end processing logic; wherein the sets of statistics dataacquired by the front-end statistics processing unit are stored in thememory device and utilized by the image pipeline processing logic toconvert the raw image data into at least one of an RGB image or a lumaimage; wherein the front-end statistics processing unit comprises:defective pixel detection and correction logic configured to detect andcorrect defective pixels in the raw image data; lens shading correctionlogic configured to correct for lens shading error in the raw image dataafter the defective pixels are detected and corrected by the defectivepixel detection and correction logic; and statistics collection logicconfigured to acquire the sets of statistics data after the lens shadingerror in the raw image data is corrected by the lens shading correctionlogic; and wherein the first front-end statistics processing unitcomprises: black level compensation logic configured to apply blacklevel compensation to the raw image data after the defective pixels aredetected and corrected by the defective pixel detection and correctionlogic, but before the lens shading error is corrected by the lensshading correction logic; and inverse black level compensation logicconfigured to apply inverse black level compensation to the raw imagedata after the lens shading error is corrected by the lens shadingcorrection logic, but before the sets of statistics data are acquired bythe statistics collection logic.
 11. The image processing system ofclaim 10, wherein each set of statistics data comprises at least one ofauto-exposure statistics, auto-white balance statistics, flickerdetection statistics, and auto-focus statistics, or any combinationthereof, pertaining to a respective frame of the raw image data.
 12. Theimage processing system of claim 10, wherein the statistics collectionlogic comprises a memory interface by which the sets of statistics datamay be written to the memory device.
 13. An imaging processing systemcomprising: a memory device; and front-end processing logic comprising:a front-end statistics processing unit configured to receive raw imagedata acquired using an image sensor, to process the raw image data toacquire a set of statistics for each frame of the raw image dataprocessed by the front-end statistics processing unit; and a front-endpixel processing unit configured to process the raw image data for atleast one of reducing noise in the raw image data or scaling the rawimage data and to provide the raw image data processed by the front-endpixel processing unit to image pipeline processing logic coupleddownstream from the front-end processing logic; wherein the sets ofstatistics data acquired by the front-end statistics processing unit arestored in the memory device and utilized by the image pipelineprocessing logic to convert the raw image data into at least one of anRGB image or a luma image; wherein the front-end pixel processing unitis configured to reduce noise in the raw image data by applying atemporal filter to the raw image data.
 14. An imaging processing systemcomprising: a memory device; and front-end processing logic comprising:a front-end statistics processing unit configured to receive raw imagedata acquired using an image sensor, to process the raw image data toacquire a set of statistics for each frame of the raw image dataprocessed by the front-end statistics processing unit; and a front-endpixel processing unit configured to process the raw image data for atleast one of reducing noise in the raw image data or scaling the rawimage data and to provide the raw image data processed by the front-endpixel processing unit to image pipeline processing logic coupleddownstream from the front-end processing logic; wherein the sets ofstatistics data acquired by the front-end statistics processing unit arestored in the memory device and utilized by the image pipelineprocessing logic to convert the raw image data into at least one of anRGB image or a luma image; wherein the front-end pixel processing unitis configured to scale the raw image data by applying a binningcompensation filter utilizing polyphase filters in horizontal andvertical directions.
 15. The image processing system of claim 14,wherein the one or more sets of statistics is used to convert the rawimage data into at least one of color image data and luma and chromaimage data.
 16. One or more tangible computer-readable storage mediahaving instructions encoded thereon for execution by a processor, theinstructions comprising: code to detect defective pixels in raw imagedata acquired using an image sensor; code to correct the detecteddefective pixels; code to correct lens shading error in the raw imagedata; and code to determine one or more sets of statistics correspondingto one or more image properties of the raw image data; wherein the codeto correcting the lens shading error in the raw image data comprises:code to determine a color component of a current input pixel of the rawimage data; code to determine the location of the current input pixelrelative to a gain grid comprising a plurality of grid points, whereineach of the plurality of grid points is associated with a lens shadinggain selected based upon the color of the current input pixel; code toapplying a lens shading gain associated with a grid point to the currentinput pixel if the location of the current input pixel correspondsdirectly to the grid point; and code to identifying a nearest set ofgrid points that encloses the current input pixel, determining a lensshading gain by interpolating the lens shading gains associated witheach of the identified set of grid points, and applying the interpolatedlens shading gain to the current input pixel if the location of theinput pixel does not correspond directly to one of the plurality of gridpoints.
 17. The one or more tangible computer-readable storage media ofclaim 16, wherein the code to correct the lens shading error is executedafter the code to correct the defective pixels.
 18. The one or moretangible computer-readable storage media of claim 17, wherein the codeto determine the one or more sets of statistics is executed after thecode to correct the lens shading error.
 19. The one or more tangiblecomputer-readable storage media of claim 16, wherein the gain gridcomprises a radial gain grid, and wherein the code to interpolate thelens shading gain uses bi-linear interpolation of the lens shading gainsassociated with each of the identified set of grid points that enclosethe input pixel in radial angular directions.
 20. An electronic devicecomprising: a memory device; a first image sensor comprising a firstsensor interface and configured to acquire a first set of raw imagedata; and an image processing system comprising: front-end imageprocessing logic comprising a first front-end statistics processing unitconfigured to receive the first set of raw image data acquired by thefirst image sensor and to acquire a first set of statistics for eachframe of the first set of raw image data processed by the firstfront-end statistics processing unit, wherein the first front-endstatistics processing unit comprises: a first defective pixel detectionand correction logic configured to detect and correct defective pixelsin the first set of raw image data; a first lens shading correctionlogic configured to correct for lens shading error in the first set ofraw image data after the defective pixels are detected and corrected bythe first defective pixel detection and correction logic; and a firststatistics collection logic configured to acquire the first sets ofstatistics data after the lens shading error in the first set of rawimage data is corrected by the first lens shading correction logic andto write the first sets of statistics data to the memory device; and adisplay device configured to display an image corresponding to the firstset of raw image data acquired by the first image sensor after the firstset of raw image data is processed by the image processing system;wherein the front-end image processing logic comprises: a second imagesensor comprising a second sensor interface and configured to acquire asecond set of raw image data; a second front-end statistics processingunit configured to receive the second raw image data acquired by thesecond image sensor and to acquire a second set of statistics for eachframe of the raw image data processed by the second front-end statisticsprocessing unit, wherein the second front-end statistics processing unitcomprises: a second defective pixel detection and correction logicconfigured to detect and correct defective pixels in the second set ofraw image data; a second lens shading correction logic configured tocorrect for lens shading error in the second set of raw image data afterthe defective pixels are detected and corrected by the second defectivepixel detection and correction logic; and a second statistics collectionlogic configured to acquire the second sets of statistics data after thelens shading error in the second set of raw image data is corrected bythe second lens shading correction logic and to write the second sets ofstatistics data to the memory device; wherein the display device isconfigured to display an image corresponding to the second set of rawimage data acquired by the second image sensor after the second set ofraw image data is processed by the image processing system.
 21. Theelectronic device of claim 20, wherein the first image sensor and thesecond image sensors are each configured to provide the first set rawimage data and the second set of raw image data, respectively, eitherdirectly to the memory device or directly to the first and secondfront-end statistics processing units, respectively, using the first andsecond sensor interfaces, respectively.
 22. The electronic device ofclaim 20, wherein the first sensor interface and the second sensorinterface comprises a Mobile Industry Processor Interface (MIPI), aStandard Mobile Imaging Architecture (SMIA) interface, or somecombination thereof.
 23. The electronic device of claim 20, wherein thedisplay device comprises an LED display, an LCD display, plasma display,OLED display, or some combination thereof.
 24. The electronic device ofclaim 23, wherein the display device comprises a touchscreen interface.25. The electronic device of claim 20, wherein the memory devicecomprises a memory device having a direct memory access (DMA) interface.26. An electronic device comprising: a memory device; a first imagesensor comprising a first sensor interface and configured to acquire afirst set of raw image data; and an image processing system comprising:front-end image processing logic comprising a first front-end statisticsprocessing unit configured to receive the first set of raw image dataacquired by the first image sensor and to acquire a first set ofstatistics for each frame of the first set of raw image data processedby the first front-end statistics processing unit, wherein the firstfront-end statistics processing unit comprises: a first defective pixeldetection and correction logic configured to detect and correctdefective pixels in the first set of raw image data; a first lensshading correction logic configured to correct for lens shading error inthe first set of raw image data after the defective pixels are detectedand corrected by the first defective pixel detection and correctionlogic; and a first statistics collection logic configured to acquire thefirst sets of statistics data after the lens shading error in the firstset of raw image data is corrected by the first lens shading correctionlogic and to write the first sets of statistics data to the memorydevice; and a display device configured to display an imagecorresponding to the first set of raw image data acquired by the firstimage sensor after the first set of raw image data is processed by theimage processing system; comprising a selection circuit configured toselect an input image signal from the memory device or from the firstsensor interface, and to provide the selected input image signal to thefirst front-end statistics processing unit.
 27. An electronic devicecomprising: a memory device; a first image sensor comprising a firstsensor interface and configured to acquire a first set of raw imagedata; and an image processing system comprising: front-end imageprocessing logic comprising a first front-end statistics processing unitconfigured to receive the first set of raw image data acquired by thefirst image sensor and to acquire a first set of statistics for eachframe of the first set of raw image data processed by the firstfront-end statistics processing unit, wherein the first front-endstatistics processing unit comprises: a first defective pixel detectionand correction logic configured to detect and correct defective pixelsin the first set of raw image data; a first lens shading correctionlogic configured to correct for lens shading error in the first set ofraw image data after the defective pixels are detected and corrected bythe first defective pixel detection and correction logic; and a firststatistics collection logic configured to acquire the first sets ofstatistics data after the lens shading error in the first set of rawimage data is corrected by the first lens shading correction logic andto write the first sets of statistics data to the memory device; and adisplay device configured to display an image corresponding to the firstset of raw image data acquired by the first image sensor after the firstset of raw image data is processed by the image processing system;wherein the electronic device comprises a desktop computer, a laptopcomputer, a tablet computer, a mobile cellular telephone, a personaldigital assistant, a portable digital media player, or some combinationthereof.